public inbox for [email protected]
help / color / mirror / Atom feedRe: Pgoutput not capturing the generated columns
60+ messages / 6 participants
[nested] [flat]
* Re: Pgoutput not capturing the generated columns
@ 2024-06-20 03:33 Peter Smith <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Peter Smith @ 2024-06-20 03:33 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected]; [email protected]
Hi, here are my review comments for v8-0001.
======
Commit message.
1.
It seems like the patch name was accidentally omitted, so it became a
mess when it defaulted to the 1st paragraph of the commit message.
======
contrib/test_decoding/test_decoding.c
2.
+ data->include_generated_columns = true;
I previously posted a comment [1, #4] that this should default to
false; IMO it is unintuitive for the test_decoding to have an
*opposite* default behaviour compared to CREATE SUBSCRIPTION.
======
doc/src/sgml/ref/create_subscription.sgml
NITPICK - remove the inconsistent blank line in SGML
======
src/backend/commands/subscriptioncmds.c
3.
+#define SUBOPT_include_generated_columns 0x00010000
I previously posted a comment [1, #6] that this should be UPPERCASE,
but it is not yet fixed.
======
src/bin/psql/describe.c
NITPICK - move and reword the bogus comment
~
4.
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("include_generated_columns"));
4a.
For consistency with every other parameter, that column title should
be written in words "Include generated columns" (not
"include_generated_columns").
~
4b.
IMO this new column belongs with the other subscription parameter
columns (e.g. put it ahead of the "Conninfo" column).
======
src/test/subscription/t/011_generated.pl
NITPICK - fixed a comment
5.
IMO, it would be better for readability if all the matching CREATE
TABLE for publisher and subscriber are kept together, instead of the
current code which is creating all publisher tables and then creating
all subscriber tables.
~~~
6.
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
+is( $result, qq(4|8
+5|10), 'confirm generated columns ARE replicated when the
subscriber-side column is not generated');
+
...
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
+is( $result, qq(4|24
+5|25), 'confirm generated columns are NOT replicated when the
subscriber-side column is also generated');
+
6a.
These SELECT all need ORDER BY to protect against the SELECT *
returning rows in some unexpected order.
~
6b.
IMO there should be more comments here to explain how you can tell the
column was NOT replicated. E.g. it is because the result value of 'b'
is the subscriber-side computed value (which you made deliberately
different to the publisher-side computed value).
======
99.
Please also refer to the attached nitpicks top-up patch for minor
cosmetic stuff.
======
[1] https://www.postgresql.org/message-id/CAHv8RjLeZtTeXpFdoY6xCPO41HtuOPMSSZgshVdb%2BV%3Dp2YHL8Q%40mail...
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index e8779dc..ee27a58 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -437,7 +437,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
-
<para>
If the subscriber-side column is also a generated column then this option
has no effect; the subscriber column will be filled as normal with the
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 75a52ce..663015d 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6609,12 +6609,12 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
+
+ /* include_generated_columns is only supported in v18 and higher */
if (pset.sversion >= 170000)
appendPQExpBuffer(&buf,
", subincludegencols AS \"%s\"\n",
gettext_noop("include_generated_columns"));
-
- // include_generated_columns
}
/* Only display subscriptions in current database. */
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 92b3dbf..cbd5015 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -24,7 +24,7 @@ $node_publisher->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-gnerated col 'b'.
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
Attachments:
[text/plain] PS_NITPICKS_20240620_v80001.txt (2.0K, ../../CAHut+PujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA@mail.gmail.com/2-PS_NITPICKS_20240620_v80001.txt)
download | inline diff:
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index e8779dc..ee27a58 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -437,7 +437,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
-
<para>
If the subscriber-side column is also a generated column then this option
has no effect; the subscriber column will be filled as normal with the
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 75a52ce..663015d 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6609,12 +6609,12 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
+
+ /* include_generated_columns is only supported in v18 and higher */
if (pset.sversion >= 170000)
appendPQExpBuffer(&buf,
", subincludegencols AS \"%s\"\n",
gettext_noop("include_generated_columns"));
-
- // include_generated_columns
}
/* Only display subscriptions in current database. */
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 92b3dbf..cbd5015 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -24,7 +24,7 @@ $node_publisher->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-gnerated col 'b'.
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-21 02:52 Peter Smith <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-06-21 02:52 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected]; [email protected]
Hi Shubham, here are some more patch v8-0001 comments that I missed yesterday.
======
src/test/subscription/t/011_generated.pl
1.
Are the PRIMARY KEY qualifiers needed for the new tab2, tab3 tables? I
don't think so.
~~~
2.
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
+is( $result, qq(4|8
+5|10), 'confirm generated columns ARE replicated when the
subscriber-side column is not generated');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub3');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
+is( $result, qq(4|24
+5|25), 'confirm generated columns are NOT replicated when the
subscriber-side column is also generated');
+
It would be prudent to do explicit "SELECT a,b" instead of "SELECT *",
for readability and to avoid any surprises.
======
Kind Regards,
Peter Smith.
Fujitsu Australia.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-22 18:06 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-06-22 18:06 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected]; [email protected]
On Thu, Jun 20, 2024 at 9:03 AM Peter Smith <[email protected]> wrote:
>
> Hi, here are my review comments for v8-0001.
>
> ======
> Commit message.
>
> 1.
> It seems like the patch name was accidentally omitted, so it became a
> mess when it defaulted to the 1st paragraph of the commit message.
>
> ======
> contrib/test_decoding/test_decoding.c
>
> 2.
> + data->include_generated_columns = true;
>
> I previously posted a comment [1, #4] that this should default to
> false; IMO it is unintuitive for the test_decoding to have an
> *opposite* default behaviour compared to CREATE SUBSCRIPTION.
>
> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> NITPICK - remove the inconsistent blank line in SGML
>
> ======
> src/backend/commands/subscriptioncmds.c
>
> 3.
> +#define SUBOPT_include_generated_columns 0x00010000
>
> I previously posted a comment [1, #6] that this should be UPPERCASE,
> but it is not yet fixed.
>
> ======
> src/bin/psql/describe.c
>
> NITPICK - move and reword the bogus comment
>
> ~
>
> 4.
> + if (pset.sversion >= 170000)
> + appendPQExpBuffer(&buf,
> + ", subincludegencols AS \"%s\"\n",
> + gettext_noop("include_generated_columns"));
>
> 4a.
> For consistency with every other parameter, that column title should
> be written in words "Include generated columns" (not
> "include_generated_columns").
>
> ~
>
> 4b.
> IMO this new column belongs with the other subscription parameter
> columns (e.g. put it ahead of the "Conninfo" column).
>
> ======
> src/test/subscription/t/011_generated.pl
>
> NITPICK - fixed a comment
>
> 5.
> IMO, it would be better for readability if all the matching CREATE
> TABLE for publisher and subscriber are kept together, instead of the
> current code which is creating all publisher tables and then creating
> all subscriber tables.
>
> ~~~
>
> 6.
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
> +is( $result, qq(4|8
> +5|10), 'confirm generated columns ARE replicated when the
> subscriber-side column is not generated');
> +
> ...
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
> +is( $result, qq(4|24
> +5|25), 'confirm generated columns are NOT replicated when the
> subscriber-side column is also generated');
> +
>
> 6a.
> These SELECT all need ORDER BY to protect against the SELECT *
> returning rows in some unexpected order.
>
> ~
>
> 6b.
> IMO there should be more comments here to explain how you can tell the
> column was NOT replicated. E.g. it is because the result value of 'b'
> is the subscriber-side computed value (which you made deliberately
> different to the publisher-side computed value).
>
> ======
>
> 99.
> Please also refer to the attached nitpicks top-up patch for minor
> cosmetic stuff.
All the comments are handled.
The attached Patch contains all the suggested changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v9-0001-Currently-generated-column-values-are-not-replica.patch (87.1K, ../../CAHv8Rj+6kwOGmn5MsRaTmciJDxtvNsyszMoPXV62OGPGzkxrCg@mail.gmail.com/2-v9-0001-Currently-generated-column-values-are-not-replica.patch)
download | inline diff:
From 9d6cc9461275459571fb276f31f027eaaaebd78c Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Mon, 20 May 2024 10:58:31 +0530
Subject: [PATCH v9] Currently generated column values are not replicated
because it is assumed that the corresponding subscriber-side table will
generate its own values for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
.../expected/decoding_into_rel.out | 39 +++++
.../test_decoding/sql/decoding_into_rel.sql | 15 +-
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/protocol.sgml | 12 ++
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 42 +++--
src/bin/pg_dump/pg_dump.c | 23 ++-
src/bin/pg_dump/pg_dump.h | 2 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 1 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 158 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 6 +
src/test/subscription/t/011_generated.pl | 63 ++++++-
src/test/subscription/t/031_column_list.pl | 4 +-
27 files changed, 410 insertions(+), 140 deletions(-)
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index 8fd3390066..94a3741408 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -103,6 +103,45 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(14 rows)
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1
+ table public.gencoltable: INSERT: a[integer]:2
+ table public.gencoltable: INSERT: a[integer]:3
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4
+ table public.gencoltable: INSERT: a[integer]:5
+ table public.gencoltable: INSERT: a[integer]:6
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
?column?
----------
diff --git a/contrib/test_decoding/sql/decoding_into_rel.sql b/contrib/test_decoding/sql/decoding_into_rel.sql
index 1068cec588..85584531a9 100644
--- a/contrib/test_decoding/sql/decoding_into_rel.sql
+++ b/contrib/test_decoding/sql/decoding_into_rel.sql
@@ -39,4 +39,17 @@ SELECT * FROM slot_changes_wrapper('regression_slot');
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..aa7690b58e 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = false;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = false;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns );
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..7a5637c5f3 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,18 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ The default is false.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..59124060d3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not supported
+ * yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 02f12f2921..75e7695353 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 95c09c9516..7405eb3deb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..663202832d 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d2b35cfb96..26796d4f9e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1413,7 +1431,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
ReorderBufferChangeType action = change->action;
TupleTableSlot *old_slot = NULL;
TupleTableSlot *new_slot = NULL;
-
if (!is_publishable_relation(relation))
return;
@@ -1531,15 +1548,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..1fb19f5c9e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4769,7 +4770,6 @@ getSubscriptions(Archive *fout)
" s.subowner,\n"
" s.subconninfo, s.subslotname, s.subsynccommit,\n"
" s.subpublications,\n");
-
if (fout->remoteVersion >= 140000)
appendPQExpBufferStr(query, " s.subbinary,\n");
else
@@ -4804,18 +4804,23 @@ getSubscriptions(Archive *fout)
if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
- " s.subenabled,\n");
+ " s.subenabled,\n");
else
appendPQExpBufferStr(query, " NULL AS suboriginremotelsn,\n"
" false AS subenabled,\n");
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
-
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols,\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4859,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4906,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5100,7 +5108,7 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
/* Build list of quoted publications and append them to query. */
if (!parsePGArray(subinfo->subpublications, &pubnames, &npubnames))
- pg_fatal("could not parse %s array", "subpublications");
+ pg_fatal("could not parse %s array", "subpublications");
publications = createPQExpBuffer();
for (i = 0; i < npubnames; i++)
@@ -5146,6 +5154,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..a2c35fe919 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,8 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
+
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..491fcb991f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6604,6 +6604,12 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Synchronous commit"),
gettext_noop("Conninfo"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
/* Skip LSN is only supported in v15 and higher */
if (pset.sversion >= 150000)
appendPQExpBuffer(&buf,
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..d9b20fb95c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +159,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* publish generated column data */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..8f3554856c 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,7 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* publish generated columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..05978c789f 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +122,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +151,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +163,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +182,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | f | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +194,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | f | 0/0
(1 row)
BEGIN;
@@ -223,10 +229,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+---------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | f | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +261,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +285,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +338,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +377,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +389,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +402,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +418,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Include generated columns | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+---------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | f | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..dbf064474c 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,6 +60,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
-- fail
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..e612970f7a 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,16 +28,50 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int)"
+);
+
+# publisher-side tab3 has generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
-);
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+ );
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+ );
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +81,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is( $result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is( $result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,6 +102,23 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub2');
+
+# the column was NOT replicated because the result value of 'b'is the subscriber-side computed value
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10), 'confirm generated columns ARE replicated when the subscriber-side column is not generated');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub3');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25), 'confirm generated columns are NOT replicated when the subscriber-side column is also generated');
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..6e73f892e9 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,7 +1202,7 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
# publication without any column (aka all columns as part of the columns
# list) are considered to have the same column list.
@@ -1211,7 +1211,7 @@ $node_publisher->safe_psql(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-22 18:08 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-06-22 18:08 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected]; [email protected]
On Fri, Jun 21, 2024 at 8:23 AM Peter Smith <[email protected]> wrote:
>
> Hi Shubham, here are some more patch v8-0001 comments that I missed yesterday.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> 1.
> Are the PRIMARY KEY qualifiers needed for the new tab2, tab3 tables? I
> don't think so.
>
> ~~~
>
> 2.
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
> +is( $result, qq(4|8
> +5|10), 'confirm generated columns ARE replicated when the
> subscriber-side column is not generated');
> +
> +$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
> +
> +$node_publisher->wait_for_catchup('sub3');
> +
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
> +is( $result, qq(4|24
> +5|25), 'confirm generated columns are NOT replicated when the
> subscriber-side column is also generated');
> +
>
> It would be prudent to do explicit "SELECT a,b" instead of "SELECT *",
> for readability and to avoid any surprises.
Both the comments are handled.
Patch v9-0001 contains all the changes required. See [1] for the changes added.
[1] https://www.postgresql.org/message-id/CAHv8Rj%2B6kwOGmn5MsRaTmciJDxtvNsyszMoPXV62OGPGzkxrCg%40mail.g...
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 60+ messages in thread
* RE: Pgoutput not capturing the generated columns
@ 2024-06-23 04:58 Hayato Kuroda (Fujitsu) <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-06-23 04:58 UTC (permalink / raw)
To: 'Shubham Khanna' <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Peter Smith <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham,
Thanks for sharing new patch! You shared as v9, but it should be v10, right?
Also, since there are no commitfest entry, I registered [1]. You can rename the
title based on the needs. Currently CFbot said OK.
Anyway, below are my comments.
01. General
Your patch contains unnecessary changes. Please remove all of them. E.g.,
```
" s.subpublications,\n");
-
```
And
```
appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
- " s.subenabled,\n");
+ " s.subenabled,\n");
```
02. General
Again, please run the pgindent/pgperltidy.
03. test_decoding
Previously I suggested to the default value of to be include_generated_columns
should be true, so you modified like that. However, Peter suggested opposite
opinion [3] and you just revised accordingly. I think either way might be okay, but
at least you must clarify the reason why you preferred to set default to false and
changed accordingly.
04. decoding_into_rel.sql
According to the comment atop this file, this test should insert result to a table.
But added case does not - we should put them at another place. I.e., create another
file.
05. decoding_into_rel.sql
```
+-- when 'include-generated-columns' is not set
```
Can you clarify the expected behavior as a comment?
06. getSubscriptions
```
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols,\n");
```
I think the comma is not needed.
Also, this error meant that you did not test to use pg_dump for instances prior PG16.
Please verify whether we can dump subscriptions and restore them accordingly.
[1]: https://commitfest.postgresql.org/48/5068/
[2]: https://www.postgresql.org/message-id/OSBPR01MB25529997E012DEABA8E15A02F5E52%40OSBPR01MB2552.jpnprd0...
[3]: https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.g...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-24 02:51 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-06-24 02:51 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected]; [email protected]
Hi, here are some patch v9-0001 comments.
I saw Kuroda-san has already posted comments for this patch so there
may be some duplication here.
======
GENERAL
1.
The later patches 0002 etc are checking to support only STORED
gencols. But, doesn't that restriction belong in this patch 0001 so
VIRTUAL columns are not decoded etc in the first place... (??)
~~~
2.
The "Generated Columns" docs mentioned in my previous review comment
[2] should be modified by this 0001 patch.
~~~
3.
I think the "Message Format" page mentioned in my previous review
comment [3] should be modified by this 0001 patch.
======
Commit message
4.
The patch name is still broken as previously mentioned [1, #1]
======
doc/src/sgml/protocol.sgml
5.
Should this docs be referring to STORED generated columns, instead of
just generated columns?
======
doc/src/sgml/ref/create_subscription.sgml
6.
Should this be docs referring to STORED generated columns, instead of
just generated columns?
======
src/bin/pg_dump/pg_dump.c
getSubscriptions:
NITPICK - tabs
NITPICK - patch removed a blank line it should not be touching
NITPICK = patch altered indents it should not be touching
NITPICK - a missing blank line that was previously present
7.
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols,\n");
There is an unwanted comma here.
~
dumpSubscription
NITPICK - patch altered indents it should not be touching
======
src/bin/pg_dump/pg_dump.h
NITPICK - unnecessary blank line
======
src/bin/psql/describe.c
describeSubscriptions
NITPICK - bad indentation
8.
In my previous review [1, #4b] I suggested this new column should be
in a different order (e.g. adjacent to the other ones ahead of
'Conninfo'), but this is not yet addressed.
======
src/test/subscription/t/011_generated.pl
NITPICK - missing space in comment
NITPICK - misleading "because" wording in the comment
======
99.
See also my attached nitpicks diff, for cosmetic issues. Please apply
whatever you agree with.
======
[1] My v8-0001 review -
https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.g...
[2] https://www.postgresql.org/message-id/CAHut%2BPvsRWq9t2tEErt5ZWZCVpNFVZjfZ_owqfdjOhh4yXb_3Q%40mail.g...
[3] https://www.postgresql.org/message-id/CAHut%2BPsHsT3V1wQ5uoH9ynbmWn4ZQqOe34X%2Bg37LSi7sgE_i2g%40mail...
Kind Regards,
Peter Smith.
Fujitsu Australia.
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 1fb19f5..9f2cac9 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,7 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
- int i_subincludegencols;
+ int i_subincludegencols;
int i,
ntups;
@@ -4770,6 +4770,7 @@ getSubscriptions(Archive *fout)
" s.subowner,\n"
" s.subconninfo, s.subslotname, s.subsynccommit,\n"
" s.subpublications,\n");
+
if (fout->remoteVersion >= 140000)
appendPQExpBufferStr(query, " s.subbinary,\n");
else
@@ -4804,7 +4805,7 @@ getSubscriptions(Archive *fout)
if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
- " s.subenabled,\n");
+ " s.subenabled,\n");
else
appendPQExpBufferStr(query, " NULL AS suboriginremotelsn,\n"
" false AS subenabled,\n");
@@ -4815,12 +4816,14 @@ getSubscriptions(Archive *fout)
else
appendPQExpBuffer(query,
" false AS subfailover,\n");
+
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
" s.subincludegencols\n");
else
appendPQExpBufferStr(query,
" false AS subincludegencols,\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index a2c35fe..8c07933 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -672,7 +672,6 @@ typedef struct _SubscriptionInfo
char *suboriginremotelsn;
char *subfailover;
char *subincludegencols;
-
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 491fcb9..00f3131 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6604,7 +6604,7 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Synchronous commit"),
gettext_noop("Conninfo"));
- /* include_generated_columns is only supported in v18 and higher */
+ /* include_generated_columns is only supported in v18 and higher */
if (pset.sversion >= 170000)
appendPQExpBuffer(&buf,
", subincludegencols AS \"%s\"\n",
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index e612970..6c8d6ce 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -106,7 +106,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
-# the column was NOT replicated because the result value of 'b'is the subscriber-side computed value
+# the column was NOT replicated (the result value of 'b' is the subscriber-side computed value)
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
5|10), 'confirm generated columns ARE replicated when the subscriber-side column is not generated');
Attachments:
[text/plain] PS_NITPICKS_20240624_v90001.txt (3.0K, ../../CAHut+PvQ8CLq-JysTTeRj4u5SC9vTVcx3AgwTHcPUEOh-UnKcQ@mail.gmail.com/2-PS_NITPICKS_20240624_v90001.txt)
download | inline diff:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 1fb19f5..9f2cac9 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,7 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
- int i_subincludegencols;
+ int i_subincludegencols;
int i,
ntups;
@@ -4770,6 +4770,7 @@ getSubscriptions(Archive *fout)
" s.subowner,\n"
" s.subconninfo, s.subslotname, s.subsynccommit,\n"
" s.subpublications,\n");
+
if (fout->remoteVersion >= 140000)
appendPQExpBufferStr(query, " s.subbinary,\n");
else
@@ -4804,7 +4805,7 @@ getSubscriptions(Archive *fout)
if (dopt->binary_upgrade && fout->remoteVersion >= 170000)
appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
- " s.subenabled,\n");
+ " s.subenabled,\n");
else
appendPQExpBufferStr(query, " NULL AS suboriginremotelsn,\n"
" false AS subenabled,\n");
@@ -4815,12 +4816,14 @@ getSubscriptions(Archive *fout)
else
appendPQExpBuffer(query,
" false AS subfailover,\n");
+
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
" s.subincludegencols\n");
else
appendPQExpBufferStr(query,
" false AS subincludegencols,\n");
+
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index a2c35fe..8c07933 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -672,7 +672,6 @@ typedef struct _SubscriptionInfo
char *suboriginremotelsn;
char *subfailover;
char *subincludegencols;
-
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 491fcb9..00f3131 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6604,7 +6604,7 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Synchronous commit"),
gettext_noop("Conninfo"));
- /* include_generated_columns is only supported in v18 and higher */
+ /* include_generated_columns is only supported in v18 and higher */
if (pset.sversion >= 170000)
appendPQExpBuffer(&buf,
", subincludegencols AS \"%s\"\n",
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index e612970..6c8d6ce 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -106,7 +106,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
-# the column was NOT replicated because the result value of 'b'is the subscriber-side computed value
+# the column was NOT replicated (the result value of 'b' is the subscriber-side computed value)
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
5|10), 'confirm generated columns ARE replicated when the subscriber-side column is not generated');
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-26 12:12 Shubham Khanna <[email protected]>
parent: Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-06-26 12:12 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Peter Smith <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Sun, Jun 23, 2024 at 10:28 AM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Hi Shubham,
>
> Thanks for sharing new patch! You shared as v9, but it should be v10, right?
> Also, since there are no commitfest entry, I registered [1]. You can rename the
> title based on the needs. Currently CFbot said OK.
>
> Anyway, below are my comments.
>
> 01. General
> Your patch contains unnecessary changes. Please remove all of them. E.g.,
>
> ```
> " s.subpublications,\n");
> -
> ```
> And
> ```
> appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
> - " s.subenabled,\n");
> + " s.subenabled,\n");
> ```
>
> 02. General
> Again, please run the pgindent/pgperltidy.
>
> 03. test_decoding
> Previously I suggested to the default value of to be include_generated_columns
> should be true, so you modified like that. However, Peter suggested opposite
> opinion [3] and you just revised accordingly. I think either way might be okay, but
> at least you must clarify the reason why you preferred to set default to false and
> changed accordingly.
I have set the default value as true in case of test_decoding. The
reason for this is even before the new feature implementation,
generated columns were getting selected.
> 04. decoding_into_rel.sql
> According to the comment atop this file, this test should insert result to a table.
> But added case does not - we should put them at another place. I.e., create another
> file.
>
> 05. decoding_into_rel.sql
> ```
> +-- when 'include-generated-columns' is not set
> ```
> Can you clarify the expected behavior as a comment?
>
> 06. getSubscriptions
> ```
> + else
> + appendPQExpBufferStr(query,
> + " false AS subincludegencols,\n");
> ```
> I think the comma is not needed.
> Also, this error meant that you did not test to use pg_dump for instances prior PG16.
> Please verify whether we can dump subscriptions and restore them accordingly.
>
> [1]: https://commitfest.postgresql.org/48/5068/
> [2]: https://www.postgresql.org/message-id/OSBPR01MB25529997E012DEABA8E15A02F5E52%40OSBPR01MB2552.jpnprd0...
> [3]: https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.g...
All the comments are handled.
The attached Patches contains all the suggested changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v11-0002-Support-replication-of-generated-column-during-i.patch (20.1K, ../../CAHv8RjJpS_XDkR6OrsmMZtCBZNxeYoCdENhC0=be0rLmNvhiQw@mail.gmail.com/2-v11-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From 37c258e8dfb151fe776ef365d9b523cfc56c3f50 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Tue, 25 Jun 2024 16:32:35 +0530
Subject: [PATCH v12 2/3] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Note that we don't copy columns when the subscriber-side column is also
generated. Those will be filled as normal with the subscriber-side computed or
default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 ---
src/backend/replication/logical/relation.c | 15 ++-
src/backend/replication/logical/tablesync.c | 114 +++++++++++++++-----
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/011_generated.pl | 88 +++++++++++++++
8 files changed, 193 insertions(+), 51 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index f942b58565..408a9157ec 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..27c34059af 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
@@ -427,6 +427,19 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
continue;
}
+ /*
+ * In case 'include_generated_columns' is 'false', we should skip the
+ * check of missing attrs for generated columns.
+ * In case 'include_generated_columns' is 'true', we should check if
+ * corresponding column for the generated column in publication column
+ * list is present in the subscription table.
+ */
+ if (!MySubscription->includegencols && attr->attgenerated)
+ {
+ entry->attrmap->attnums[i] = -1;
+ continue;
+ }
+
attnum = logicalrep_rel_att_by_name(remoterel,
NameStr(attr->attname));
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b00267f042..b3fde6afb3 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -692,20 +693,63 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create list of columns for COPY based on logical relation mapping. Do not
+ * include generated columns of the subscription table in the column list.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *gencollist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ gencollist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
+
+ /* Loop to handle subscription table generated columns. */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if(!remotegenlist[attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'gencollist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ gencollist[attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if(!gencollist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
return attnamelist;
}
@@ -791,16 +835,17 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist_res;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
@@ -948,18 +993,24 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey),"
+ " a.attgenerated != ''"
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
+ walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1024,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist_res = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1057,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist_res[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1069,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist = remotegenlist_res;
walrcv_clear_result(res);
/*
@@ -1123,10 +1177,12 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1191,17 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Regular table with no row filter and 'include_generated_columns'
+ * specified as 'false' during creation of subscription.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL &&
+ !MySubscription->includegencols)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1169,17 +1231,20 @@ copy_table(Relation rel)
else
{
/*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
+ * For non-tables and tables with row filters and when
+ * 'include_generated_columns' is specified as 'true', we need to do
+ * COPY (SELECT ...), as normal COPY of generated column is not
+ * supported. For tables with any row filters, build a SELECT query
+ * with OR'ed row filters for COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_ptr(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1302,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index b78e3c6d6a..d7c4298377 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,9 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index dbf064474c..838881be50 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,9 +60,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index bc6033adb0..3ab004429f 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -16,6 +16,8 @@ $node_publisher->start;
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 10");
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -41,6 +43,28 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+# tab4: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int , b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab5: publisher-side non-generated col 'b' --> subscriber-side generated col 'b'
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# tab6: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# columns on subscriber in different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)");
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab6 (c int GENERATED ALWAYS AS (a * 22) STORED, b int, a int)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -49,6 +73,12 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab5 (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR TABLE tab1");
@@ -56,6 +86,12 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub2 FOR TABLE tab2");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub4 FOR TABLE tab4");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub5 FOR TABLE tab5");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub6 FOR TABLE tab6");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
@@ -69,6 +105,14 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr' PUBLICATION pub4 WITH (include_generated_columns = true)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub6 CONNECTION '$publisher_connstr' PUBLICATION pub6 WITH (include_generated_columns = true)"
+);
+
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -121,6 +165,50 @@ is( $result, qq(4|24
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub4');
+
+# gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
+# gen-col 'c' in publisher not replicating to gen-col 'c' on subscriber
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated column with initial sync');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab6 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub6');
+
+# gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
+# order of column is different on subscriber
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated column with initial sync different column order');
+
+# NOT gen-col 'b' in publisher not replicating to gen-col 'b' on subscriber
+my $offset = -s $node_subscriber->logfile;
+
+# sub5 will cause table sync worker to restart repetitively
+# So SUBSCRIPTION sub5 is created separately
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub5 CONNECTION '$publisher_connstr' PUBLICATION pub5 WITH (include_generated_columns = true)"
+);
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab5" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub5");
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
--
2.34.1
[application/octet-stream] v11-0001-Enable-support-for-include_generated_columns-opt.patch (90.2K, ../../CAHv8RjJpS_XDkR6OrsmMZtCBZNxeYoCdENhC0=be0rLmNvhiQw@mail.gmail.com/3-v11-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 7a9c0f6bd2f809a90a9a7ee271148837b4df5b0f Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Mon, 20 May 2024 10:58:31 +0530
Subject: [PATCH v12] Enable support for 'include_generated_columns' option in
'logical replication'
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
contrib/test_decoding/expected/binary.out | 6 +-
.../expected/decoding_into_rel.out | 6 -
.../expected/generated_columns.out | 44 +++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/decoding_into_rel.sql | 2 +-
.../test_decoding/sql/generated_columns.sql | 20 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 5 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 158 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 6 +
src/test/subscription/t/011_generated.pl | 61 ++++++-
src/test/subscription/t/031_column_list.pl | 4 +-
33 files changed, 428 insertions(+), 152 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..9ecd4fa0b7 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot
diff --git a/contrib/test_decoding/expected/binary.out b/contrib/test_decoding/expected/binary.out
index b3a3509595..c30abc7692 100644
--- a/contrib/test_decoding/expected/binary.out
+++ b/contrib/test_decoding/expected/binary.out
@@ -1,11 +1,7 @@
-- predictability
SET synchronous_commit = on;
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
- ?column?
-----------
- init
-(1 row)
-
+ERROR: replication slot "regression_slot" already exists
-- succeeds, textual plugin, textual consumer
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1');
data
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index 8fd3390066..f763e05dc7 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -103,9 +103,3 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(14 rows)
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
- ?column?
-----------
- stop
-(1 row)
-
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..3f8d6ead96
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,44 @@
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set then columns will not be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4
+ table public.gencoltable: INSERT: a[integer]:5
+ table public.gencoltable: INSERT: a[integer]:6
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+-----------
+ stop
+(1 row)
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..eab4aa68c9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/decoding_into_rel.sql b/contrib/test_decoding/sql/decoding_into_rel.sql
index 1068cec588..bcb5bb50b8 100644
--- a/contrib/test_decoding/sql/decoding_into_rel.sql
+++ b/contrib/test_decoding/sql/decoding_into_rel.sql
@@ -39,4 +39,4 @@ SELECT * FROM slot_changes_wrapper('regression_slot');
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..bb50fc1fa4
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,20 @@
+-- test that we can insert the result of a 'include_generated_columns'
+-- into the tables created. That's really not a good idea in practical terms,
+-- but provides a nice test.
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set then columns will not be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 9b71c97bdf..0f6201376e 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,9 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ 'include_generated_columns' option controls whether generated columns
+ should be included in the string representation of tuples during
+ logical decoding in PostgreSQL. The default is <literal>true</literal>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..39207a6755 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,18 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ The default is true.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6523,11 +6535,6 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</varlistentry>
</variablelist>
- <para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
- </para>
-
<variablelist>
<varlistentry>
<term>Int8</term>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..f942b58565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 02f12f2921..75e7695353 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 95c09c9516..7405eb3deb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..663202832d 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d2b35cfb96..00c6566959 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..8fdd1a6591 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5146,6 +5156,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..8c07933d09 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..59f2ce30de 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6598,6 +6598,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..ccff291b85 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* publish generated column data */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..c761c4b829 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..b78e3c6d6a 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +122,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +151,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +163,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +182,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +194,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +229,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +261,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +285,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +338,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +377,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +389,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +402,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +418,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..dbf064474c 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,6 +60,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
-- fail
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..bc6033adb0 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,17 +28,47 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
+
+# publisher-side tab3 has generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+);
+
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +77,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is($result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is($result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,6 +98,29 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub2');
+
+# the column was NOT replicated (the result value of 'b' is the subscriber-side computed value)
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+);
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub3');
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..6e73f892e9 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,7 +1202,7 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
# publication without any column (aka all columns as part of the columns
# list) are considered to have the same column list.
@@ -1211,7 +1211,7 @@ $node_publisher->safe_psql(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
[application/octet-stream] v11-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (18.7K, ../../CAHv8RjJpS_XDkR6OrsmMZtCBZNxeYoCdENhC0=be0rLmNvhiQw@mail.gmail.com/4-v11-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From 1b21e84ccfda07f57ab96371869b204b4b9b9497 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Tue, 25 Jun 2024 16:49:13 +0530
Subject: [PATCH v11] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
contrib/test_decoding/expected/binary.out | 6 +++++-
.../expected/decoding_into_rel.out | 6 ++++++
.../expected/generated_columns.out | 13 +++++++++++-
.../test_decoding/sql/generated_columns.sql | 4 +++-
contrib/test_decoding/test_decoding.c | 8 +++++++-
doc/src/sgml/ddl.sgml | 2 +-
doc/src/sgml/protocol.sgml | 8 ++++----
doc/src/sgml/ref/create_subscription.sgml | 4 ++--
src/backend/catalog/pg_publication.c | 13 +++++++++++-
src/backend/commands/subscriptioncmds.c | 14 -------------
src/backend/replication/logical/proto.c | 8 ++++----
src/backend/replication/logical/relation.c | 3 +++
src/backend/replication/logical/tablesync.c | 19 ++++++++++++++----
src/backend/replication/pgoutput/pgoutput.c | 5 ++++-
src/test/subscription/t/011_generated.pl | 20 ++++++++++---------
15 files changed, 89 insertions(+), 44 deletions(-)
diff --git a/contrib/test_decoding/expected/binary.out b/contrib/test_decoding/expected/binary.out
index c30abc7692..b3a3509595 100644
--- a/contrib/test_decoding/expected/binary.out
+++ b/contrib/test_decoding/expected/binary.out
@@ -1,7 +1,11 @@
-- predictability
SET synchronous_commit = on;
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
-ERROR: replication slot "regression_slot" already exists
+ ?column?
+----------
+ init
+(1 row)
+
-- succeeds, textual plugin, textual consumer
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1');
data
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index f763e05dc7..8fd3390066 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -103,3 +103,9 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(14 rows)
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index 3f8d6ead96..268dce1f6a 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -1,3 +1,12 @@
+-- test that we can insert the result of a 'include_generated_columns'
+-- into the tables created. That's really not a good idea in practical terms,
+-- but provides a nice test.
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
-- check include-generated-columns option with generated column
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
-- when 'include-generated-columns' is not set then columns will not be replicated
@@ -39,6 +48,8 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
?column?
------------
+----------
stop
(1 row)
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index bb50fc1fa4..9e707c5125 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -17,4 +17,6 @@ INSERT INTO gencoltable (a) VALUES (4), (5), (6);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..7aca5a19ac 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,7 +557,13 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
+ /*
+ * Don't print virtual generated column. Don't print stored
+ * generated column if 'include_generated_columns' is false.
+ *
+ * TODO: can use ATTRIBUTE_GENERATED_VIRTUAL to simpilfy
+ */
+ if (attr->attgenerated && (attr->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
/*
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index b68f275d98..0f6201376e 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -516,7 +516,7 @@ CREATE TABLE people (
<para>
'include_generated_columns' option controls whether generated columns
should be included in the string representation of tuples during
- logical decoding in PostgreSQL. The default is <litearl>true</literal>.
+ logical decoding in PostgreSQL. The default is <literal>true</literal>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 39207a6755..dd03aab60b 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3310,10 +3310,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
- The default is true.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL. The default is true.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..79ccb9bd71 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present in
+ the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..e5e5aef243 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -533,6 +534,16 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ /*
+ * TODO: simplify the expression
+ */
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated &&
+ TupleDescAttr(tupdesc, attnum - 1)->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot use virtual generated column \"%s\" in publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1225,7 +1236,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped)
+ if (att->attisdropped || (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED))
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index f942b58565..408a9157ec 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 7405eb3deb..e82e53e384 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,7 +793,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -817,7 +817,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -957,7 +957,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -981,7 +981,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 27c34059af..e1b1693700 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -427,6 +427,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
continue;
}
+ if (attr->attgenerated && attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
/*
* In case 'include_generated_columns' is 'false', we should skip the
* check of missing attrs for generated columns.
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b3fde6afb3..d44f10901e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -712,7 +712,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (!attr->attgenerated)
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
attnum = logicalrep_rel_att_by_name(&rel->remoterel,
@@ -1001,10 +1001,21 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
" WHERE a.attnum > 0::pg_catalog.int2"
" AND NOT a.attisdropped", lrel->remoteid);
- if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
- walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
- !MySubscription->includegencols)
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000)
+ {
+ bool gencols_allowed = walrcv_server_version(LogRepWorkerWalRcvConn) >= 170000
+ && MySubscription->includegencols;
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
+ {
+ /* Replication of generated cols is not supported. */
appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+ }
appendStringInfo(&cmd,
" AND a.attrelid = %u"
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 00c6566959..69aaf849e4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -784,7 +784,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -1106,6 +1106,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 3ab004429f..bb086791a3 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -30,20 +30,20 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+# publisher-side tab2 has stored generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
-# publisher-side tab3 has generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has stored generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
-# tab4: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# tab4: publisher-side stored generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and stored generated-col 'c'
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int , b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -52,19 +52,21 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
);
-# tab5: publisher-side non-generated col 'b' --> subscriber-side generated col 'b'
+# tab5: publisher-side non-generated col 'b' --> subscriber-side stored generated col 'b'
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
-# tab6: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# tab6: publisher-side stored generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and stored generated-col 'c'
# columns on subscriber in different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab6 (c int GENERATED ALWAYS AS (a * 22) STORED, b int, a int)");
+# TODO: Add tests related to replication of VIRTUAL GNERATED COLUMNS
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -169,8 +171,8 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub4');
-# gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
-# gen-col 'c' in publisher not replicating to gen-col 'c' on subscriber
+# stored gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
+# stored gen-col 'c' in publisher not replicating to stored gen-col 'c' on subscriber
$result =
$node_subscriber->safe_psql('postgres',
"SELECT a, b, c FROM tab4 ORDER BY a");
@@ -184,7 +186,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab6 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub6');
-# gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
+# stored gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
# order of column is different on subscriber
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b, c FROM tab6 ORDER BY a");
@@ -194,7 +196,7 @@ is( $result, qq(1|2|22
4|8|88
5|10|110), 'replicate generated column with initial sync different column order');
-# NOT gen-col 'b' in publisher not replicating to gen-col 'b' on subscriber
+# NOT gen-col 'b' in publisher not replicating to stored gen-col 'b' on subscriber
my $offset = -s $node_subscriber->logfile;
# sub5 will cause table sync worker to restart repetitively
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-26 12:16 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-06-26 12:16 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected]; [email protected]
On Mon, Jun 24, 2024 at 8:21 AM Peter Smith <[email protected]> wrote:
>
> Hi, here are some patch v9-0001 comments.
>
> I saw Kuroda-san has already posted comments for this patch so there
> may be some duplication here.
>
> ======
> GENERAL
>
> 1.
> The later patches 0002 etc are checking to support only STORED
> gencols. But, doesn't that restriction belong in this patch 0001 so
> VIRTUAL columns are not decoded etc in the first place... (??)
>
> ~~~
>
> 2.
> The "Generated Columns" docs mentioned in my previous review comment
> [2] should be modified by this 0001 patch.
>
> ~~~
>
> 3.
> I think the "Message Format" page mentioned in my previous review
> comment [3] should be modified by this 0001 patch.
>
> ======
> Commit message
>
> 4.
> The patch name is still broken as previously mentioned [1, #1]
>
> ======
> doc/src/sgml/protocol.sgml
>
> 5.
> Should this docs be referring to STORED generated columns, instead of
> just generated columns?
>
> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> 6.
> Should this be docs referring to STORED generated columns, instead of
> just generated columns?
>
> ======
> src/bin/pg_dump/pg_dump.c
>
> getSubscriptions:
> NITPICK - tabs
> NITPICK - patch removed a blank line it should not be touching
> NITPICK = patch altered indents it should not be touching
> NITPICK - a missing blank line that was previously present
>
> 7.
> + else
> + appendPQExpBufferStr(query,
> + " false AS subincludegencols,\n");
>
> There is an unwanted comma here.
>
> ~
>
> dumpSubscription
> NITPICK - patch altered indents it should not be touching
>
> ======
> src/bin/pg_dump/pg_dump.h
>
> NITPICK - unnecessary blank line
>
> ======
> src/bin/psql/describe.c
>
> describeSubscriptions
> NITPICK - bad indentation
>
> 8.
> In my previous review [1, #4b] I suggested this new column should be
> in a different order (e.g. adjacent to the other ones ahead of
> 'Conninfo'), but this is not yet addressed.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> NITPICK - missing space in comment
> NITPICK - misleading "because" wording in the comment
>
> ======
>
> 99.
> See also my attached nitpicks diff, for cosmetic issues. Please apply
> whatever you agree with.
>
> ======
> [1] My v8-0001 review -
> https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.g...
> [2] https://www.postgresql.org/message-id/CAHut%2BPvsRWq9t2tEErt5ZWZCVpNFVZjfZ_owqfdjOhh4yXb_3Q%40mail.g...
> [3] https://www.postgresql.org/message-id/CAHut%2BPsHsT3V1wQ5uoH9ynbmWn4ZQqOe34X%2Bg37LSi7sgE_i2g%40mail...
All the comments are handled.
I have attached the updated patch v11 here in [1]. See [1] for the
changes added.
[1] https://www.postgresql.org/message-id/CAHv8RjJpS_XDkR6OrsmMZtCBZNxeYoCdENhC0%3Dbe0rLmNvhiQw%40mail.g...
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-27 05:29 Shubham Khanna <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-06-27 05:29 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Peter Smith <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
>All the comments are handled.
>
> The attached Patches contain all the suggested changes.
v11-0003 patch was not getting applied, so here are the updated
patches for the same.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v12-0002-Support-replication-of-generated-column-during-i.patch (20.1K, ../../CAHv8RjJSQk0MNK5wwyL5pvt_dbaKnfKSKsNFsg3sfVboFpHMeA@mail.gmail.com/2-v12-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From 37c258e8dfb151fe776ef365d9b523cfc56c3f50 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Tue, 25 Jun 2024 16:32:35 +0530
Subject: [PATCH v12 2/3] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Note that we don't copy columns when the subscriber-side column is also
generated. Those will be filled as normal with the subscriber-side computed or
default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 ---
src/backend/replication/logical/relation.c | 15 ++-
src/backend/replication/logical/tablesync.c | 114 +++++++++++++++-----
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/011_generated.pl | 88 +++++++++++++++
8 files changed, 193 insertions(+), 51 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index f942b58565..408a9157ec 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..27c34059af 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
@@ -427,6 +427,19 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
continue;
}
+ /*
+ * In case 'include_generated_columns' is 'false', we should skip the
+ * check of missing attrs for generated columns.
+ * In case 'include_generated_columns' is 'true', we should check if
+ * corresponding column for the generated column in publication column
+ * list is present in the subscription table.
+ */
+ if (!MySubscription->includegencols && attr->attgenerated)
+ {
+ entry->attrmap->attnums[i] = -1;
+ continue;
+ }
+
attnum = logicalrep_rel_att_by_name(remoterel,
NameStr(attr->attname));
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b00267f042..b3fde6afb3 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -692,20 +693,63 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create list of columns for COPY based on logical relation mapping. Do not
+ * include generated columns of the subscription table in the column list.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *gencollist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ gencollist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
+
+ /* Loop to handle subscription table generated columns. */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if(!remotegenlist[attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'gencollist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ gencollist[attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if(!gencollist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
return attnamelist;
}
@@ -791,16 +835,17 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist_res;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
@@ -948,18 +993,24 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey),"
+ " a.attgenerated != ''"
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
+ walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1024,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist_res = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1057,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist_res[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1069,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist = remotegenlist_res;
walrcv_clear_result(res);
/*
@@ -1123,10 +1177,12 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1191,17 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Regular table with no row filter and 'include_generated_columns'
+ * specified as 'false' during creation of subscription.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL &&
+ !MySubscription->includegencols)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1169,17 +1231,20 @@ copy_table(Relation rel)
else
{
/*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
+ * For non-tables and tables with row filters and when
+ * 'include_generated_columns' is specified as 'true', we need to do
+ * COPY (SELECT ...), as normal COPY of generated column is not
+ * supported. For tables with any row filters, build a SELECT query
+ * with OR'ed row filters for COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_ptr(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1302,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index b78e3c6d6a..d7c4298377 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,9 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index dbf064474c..838881be50 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,9 +60,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index bc6033adb0..3ab004429f 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -16,6 +16,8 @@ $node_publisher->start;
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 10");
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -41,6 +43,28 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+# tab4: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int , b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab5: publisher-side non-generated col 'b' --> subscriber-side generated col 'b'
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# tab6: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# columns on subscriber in different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)");
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab6 (c int GENERATED ALWAYS AS (a * 22) STORED, b int, a int)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -49,6 +73,12 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab5 (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR TABLE tab1");
@@ -56,6 +86,12 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub2 FOR TABLE tab2");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub4 FOR TABLE tab4");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub5 FOR TABLE tab5");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub6 FOR TABLE tab6");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
@@ -69,6 +105,14 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr' PUBLICATION pub4 WITH (include_generated_columns = true)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub6 CONNECTION '$publisher_connstr' PUBLICATION pub6 WITH (include_generated_columns = true)"
+);
+
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -121,6 +165,50 @@ is( $result, qq(4|24
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub4');
+
+# gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
+# gen-col 'c' in publisher not replicating to gen-col 'c' on subscriber
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated column with initial sync');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab6 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub6');
+
+# gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
+# order of column is different on subscriber
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated column with initial sync different column order');
+
+# NOT gen-col 'b' in publisher not replicating to gen-col 'b' on subscriber
+my $offset = -s $node_subscriber->logfile;
+
+# sub5 will cause table sync worker to restart repetitively
+# So SUBSCRIPTION sub5 is created separately
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub5 CONNECTION '$publisher_connstr' PUBLICATION pub5 WITH (include_generated_columns = true)"
+);
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab5" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub5");
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
--
2.34.1
[application/octet-stream] v12-0001-Enable-support-for-include_generated_columns-opt.patch (90.2K, ../../CAHv8RjJSQk0MNK5wwyL5pvt_dbaKnfKSKsNFsg3sfVboFpHMeA@mail.gmail.com/3-v12-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 7a9c0f6bd2f809a90a9a7ee271148837b4df5b0f Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Mon, 20 May 2024 10:58:31 +0530
Subject: [PATCH v12] Enable support for 'include_generated_columns' option in
'logical replication'
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
contrib/test_decoding/expected/binary.out | 6 +-
.../expected/decoding_into_rel.out | 6 -
.../expected/generated_columns.out | 44 +++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/decoding_into_rel.sql | 2 +-
.../test_decoding/sql/generated_columns.sql | 20 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 5 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 158 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 6 +
src/test/subscription/t/011_generated.pl | 61 ++++++-
src/test/subscription/t/031_column_list.pl | 4 +-
33 files changed, 428 insertions(+), 152 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..9ecd4fa0b7 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot
diff --git a/contrib/test_decoding/expected/binary.out b/contrib/test_decoding/expected/binary.out
index b3a3509595..c30abc7692 100644
--- a/contrib/test_decoding/expected/binary.out
+++ b/contrib/test_decoding/expected/binary.out
@@ -1,11 +1,7 @@
-- predictability
SET synchronous_commit = on;
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
- ?column?
-----------
- init
-(1 row)
-
+ERROR: replication slot "regression_slot" already exists
-- succeeds, textual plugin, textual consumer
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1');
data
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index 8fd3390066..f763e05dc7 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -103,9 +103,3 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(14 rows)
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
- ?column?
-----------
- stop
-(1 row)
-
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..3f8d6ead96
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,44 @@
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set then columns will not be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4
+ table public.gencoltable: INSERT: a[integer]:5
+ table public.gencoltable: INSERT: a[integer]:6
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+-----------
+ stop
+(1 row)
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..eab4aa68c9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/decoding_into_rel.sql b/contrib/test_decoding/sql/decoding_into_rel.sql
index 1068cec588..bcb5bb50b8 100644
--- a/contrib/test_decoding/sql/decoding_into_rel.sql
+++ b/contrib/test_decoding/sql/decoding_into_rel.sql
@@ -39,4 +39,4 @@ SELECT * FROM slot_changes_wrapper('regression_slot');
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..bb50fc1fa4
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,20 @@
+-- test that we can insert the result of a 'include_generated_columns'
+-- into the tables created. That's really not a good idea in practical terms,
+-- but provides a nice test.
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set then columns will not be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 9b71c97bdf..0f6201376e 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,9 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ 'include_generated_columns' option controls whether generated columns
+ should be included in the string representation of tuples during
+ logical decoding in PostgreSQL. The default is <literal>true</literal>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..39207a6755 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,18 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ The default is true.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6523,11 +6535,6 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</varlistentry>
</variablelist>
- <para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
- </para>
-
<variablelist>
<varlistentry>
<term>Int8</term>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..f942b58565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 02f12f2921..75e7695353 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 95c09c9516..7405eb3deb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..663202832d 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d2b35cfb96..00c6566959 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..8fdd1a6591 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5146,6 +5156,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..8c07933d09 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..59f2ce30de 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6598,6 +6598,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..ccff291b85 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* publish generated column data */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..c761c4b829 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..b78e3c6d6a 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +122,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +151,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +163,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +182,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +194,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +229,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +261,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +285,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +338,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +377,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +389,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +402,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +418,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..dbf064474c 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,6 +60,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
-- fail
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..bc6033adb0 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,17 +28,47 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
+
+# publisher-side tab3 has generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+);
+
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +77,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is($result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is($result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,6 +98,29 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub2');
+
+# the column was NOT replicated (the result value of 'b' is the subscriber-side computed value)
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+);
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub3');
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..6e73f892e9 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,7 +1202,7 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
# publication without any column (aka all columns as part of the columns
# list) are considered to have the same column list.
@@ -1211,7 +1211,7 @@ $node_publisher->safe_psql(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
[application/octet-stream] v12-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (17.1K, ../../CAHv8RjJSQk0MNK5wwyL5pvt_dbaKnfKSKsNFsg3sfVboFpHMeA@mail.gmail.com/4-v12-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From 881eb624409497d957763ba92ffc6b34f421d02e Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Tue, 25 Jun 2024 16:49:13 +0530
Subject: [PATCH v11 3/3] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
contrib/test_decoding/expected/binary.out | 6 +++++-
.../expected/decoding_into_rel.out | 6 ++++++
.../expected/generated_columns.out | 13 +++++++++++-
.../test_decoding/sql/generated_columns.sql | 4 +++-
contrib/test_decoding/test_decoding.c | 8 +++++++-
doc/src/sgml/protocol.sgml | 8 ++++----
doc/src/sgml/ref/create_subscription.sgml | 4 ++--
src/backend/catalog/pg_publication.c | 13 +++++++++++-
src/backend/replication/logical/proto.c | 8 ++++----
src/backend/replication/logical/relation.c | 3 +++
src/backend/replication/logical/tablesync.c | 19 ++++++++++++++----
src/backend/replication/pgoutput/pgoutput.c | 5 ++++-
src/test/subscription/t/011_generated.pl | 20 ++++++++++---------
13 files changed, 88 insertions(+), 29 deletions(-)
diff --git a/contrib/test_decoding/expected/binary.out b/contrib/test_decoding/expected/binary.out
index c30abc7692..b3a3509595 100644
--- a/contrib/test_decoding/expected/binary.out
+++ b/contrib/test_decoding/expected/binary.out
@@ -1,7 +1,11 @@
-- predictability
SET synchronous_commit = on;
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
-ERROR: replication slot "regression_slot" already exists
+ ?column?
+----------
+ init
+(1 row)
+
-- succeeds, textual plugin, textual consumer
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1');
data
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index f763e05dc7..8fd3390066 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -103,3 +103,9 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(14 rows)
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index 3f8d6ead96..268dce1f6a 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -1,3 +1,12 @@
+-- test that we can insert the result of a 'include_generated_columns'
+-- into the tables created. That's really not a good idea in practical terms,
+-- but provides a nice test.
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
-- check include-generated-columns option with generated column
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
-- when 'include-generated-columns' is not set then columns will not be replicated
@@ -39,6 +48,8 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
?column?
------------
+----------
stop
(1 row)
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index bb50fc1fa4..9e707c5125 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -17,4 +17,6 @@ INSERT INTO gencoltable (a) VALUES (4), (5), (6);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..7aca5a19ac 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,7 +557,13 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
+ /*
+ * Don't print virtual generated column. Don't print stored
+ * generated column if 'include_generated_columns' is false.
+ *
+ * TODO: can use ATTRIBUTE_GENERATED_VIRTUAL to simpilfy
+ */
+ if (attr->attgenerated && (attr->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
/*
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 39207a6755..dd03aab60b 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3310,10 +3310,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
- The default is true.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL. The default is true.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..e5e5aef243 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -533,6 +534,16 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ /*
+ * TODO: simplify the expression
+ */
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated &&
+ TupleDescAttr(tupdesc, attnum - 1)->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot use virtual generated column \"%s\" in publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1225,7 +1236,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped)
+ if (att->attisdropped || (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED))
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 7405eb3deb..e82e53e384 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,7 +793,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -817,7 +817,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -957,7 +957,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -981,7 +981,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 27c34059af..e1b1693700 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -427,6 +427,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
continue;
}
+ if (attr->attgenerated && attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
/*
* In case 'include_generated_columns' is 'false', we should skip the
* check of missing attrs for generated columns.
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b3fde6afb3..d44f10901e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -712,7 +712,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (!attr->attgenerated)
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
attnum = logicalrep_rel_att_by_name(&rel->remoterel,
@@ -1001,10 +1001,21 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
" WHERE a.attnum > 0::pg_catalog.int2"
" AND NOT a.attisdropped", lrel->remoteid);
- if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
- walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
- !MySubscription->includegencols)
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000)
+ {
+ bool gencols_allowed = walrcv_server_version(LogRepWorkerWalRcvConn) >= 170000
+ && MySubscription->includegencols;
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
+ {
+ /* Replication of generated cols is not supported. */
appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+ }
appendStringInfo(&cmd,
" AND a.attrelid = %u"
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 00c6566959..69aaf849e4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -784,7 +784,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -1106,6 +1106,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 3ab004429f..bb086791a3 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -30,20 +30,20 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+# publisher-side tab2 has stored generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
-# publisher-side tab3 has generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has stored generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
-# tab4: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# tab4: publisher-side stored generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and stored generated-col 'c'
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int , b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -52,19 +52,21 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
);
-# tab5: publisher-side non-generated col 'b' --> subscriber-side generated col 'b'
+# tab5: publisher-side non-generated col 'b' --> subscriber-side stored generated col 'b'
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
-# tab6: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# tab6: publisher-side stored generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and stored generated-col 'c'
# columns on subscriber in different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab6 (c int GENERATED ALWAYS AS (a * 22) STORED, b int, a int)");
+# TODO: Add tests related to replication of VIRTUAL GNERATED COLUMNS
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -169,8 +171,8 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub4');
-# gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
-# gen-col 'c' in publisher not replicating to gen-col 'c' on subscriber
+# stored gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
+# stored gen-col 'c' in publisher not replicating to stored gen-col 'c' on subscriber
$result =
$node_subscriber->safe_psql('postgres',
"SELECT a, b, c FROM tab4 ORDER BY a");
@@ -184,7 +186,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab6 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub6');
-# gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
+# stored gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
# order of column is different on subscriber
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b, c FROM tab6 ORDER BY a");
@@ -194,7 +196,7 @@ is( $result, qq(1|2|22
4|8|88
5|10|110), 'replicate generated column with initial sync different column order');
-# NOT gen-col 'b' in publisher not replicating to gen-col 'b' on subscriber
+# NOT gen-col 'b' in publisher not replicating to stored gen-col 'b' on subscriber
my $offset = -s $node_subscriber->logfile;
# sub5 will cause table sync worker to restart repetitively
--
2.41.0.windows.3
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-06-27 09:10 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-06-27 09:10 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, here are some patch v11-0001 comments.
(BTW, I had difficulty reviewing this because something seemed strange
with the changes this patch made to the test_decoding tests).
======
General
1. Patch name
Patch name does not need to quote 'logical replication'
~
2. test_decoding tests
Multiple test_decoding tests were failing for me. There is something
very suspicious about the unexplained changes the patch made to the
expected "binary.out" and "decoding_into_rel.out" etc. I REVERTED all
those changes in my nitpicks top-up to get everything working. Please
re-confirm that all the test_decoding tests are OK!
======
Commit Message
3.
Since you are including the example usage for test_decoding, I think
it's better to include the example usage of CREATE SUBSCRIPTION also.
======
contrib/test_decoding/expected/binary.out
4.
SELECT 'init' FROM
pg_create_logical_replication_slot('regression_slot',
'test_decoding');
- ?column?
-----------
- init
-(1 row)
-
+ERROR: replication slot "regression_slot" already exists
Huh? Why is this unrelated expected output changed by this patch?
The test_decoding test fails for me unless I REVERT this change!! See
my nitpicks diff.
======
.../expected/decoding_into_rel.out
5.
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
- ?column?
-----------
- stop
-(1 row)
-
Huh? Why is this unrelated expected output changed by this patch?
The test_decoding test fails for me unless I REVERT this change!! See
my nitpicks diff.
======
.../test_decoding/sql/decoding_into_rel.sql
6.
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
Huh, Why does this patch change this code at all? I REVERTED this
change. See my nitpicks diff.
======
.../test_decoding/sql/generated_columns.sql
(see my nitpicks replacement file for this test)
7.
+-- test that we can insert the result of a 'include_generated_columns'
+-- into the tables created. That's really not a good idea in practical terms,
+-- but provides a nice test.
NITPICK - I didn't understand the point of this comment. I updated
the comment according to my understanding.
~
NITPICK - The comment "when 'include-generated-columns' is not set
then columns will not be replicated" is the opposite of what the
result is. I changed this comment.
NITPICK - modified and unified wording of some of the other comments
NITPICK - changed some blank lines
======
contrib/test_decoding/test_decoding.c
8.
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
Is there any way to test that "elem->arg == NULL" in the
generated.sql? OTOH, if it is not possible to get here then is the
code even needed?
======
doc/src/sgml/ddl.sgml
9.
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ 'include_generated_columns' option controls whether generated columns
+ should be included in the string representation of tuples during
+ logical decoding in PostgreSQL. The default is <literal>true</literal>.
</para>
NITPICK - Use proper markdown instead of single quotes for the parameter.
NITPICK - I think this can be reworded slightly to provide a
cross-reference to the CREATE SUBSCRIPTION parameter for more details
(which means then we can avoid repeating details like the default
value here). PSA my nitpicks diff for an example of how I thought docs
should look.
======
doc/src/sgml/protocol.sgml
10.
+ The default is true.
No, it isn't. AFAIK you made the default behaviour true only for
'test_decoding', but the default for CREATE SUBSCRIPTION remains
*false* because that is the existing PG17 behaviour. And the default
for the 'include_generated_columns' in the protocol is *also* false to
match the CREATE SUBSCRIPTION default.
e.g. libpqwalreceiver.c only sets ", include_generated_columns 'true'"
when options->proto.logical.include_generated_columns
e.g. worker.c says: options->proto.logical.include_generated_columns =
MySubscription->includegencols;
e.g. subscriptioncmds.c sets default: opts->include_generated_columns = false;
(This confirmed my previous review expectation that using different
default behaviours for test_decoding and pgoutput would surely lead to
confusion)
~~~
11.
- <para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
- </para>
-
AFAIK you cannot just remove this entire paragraph because I thought
it was still relevant to talking about "... the following message
part". But, if you don't want to explain and cross-reference about
'include_generated_columns' then maybe it is OK just to remove the
"(except generated columns)" part?
======
src/test/subscription/t/011_generated.pl
NITPICK - comment typo /tab2/tab3/
NITPICK - remove some blank lines
~~~
12.
# the column was NOT replicated (the result value of 'b' is the
subscriber-side computed value)
NITPICK - I think this comment is wrong for the tab2 test because here
col 'b' IS replicated. I have added much more substantial test case
comments in the attached nitpicks diff. PSA.
======
src/test/subscription/t/031_column_list.pl
13.
NITPICK - IMO there is a missing word "list" in the comment. This bug
existed already on HEAD but since this patch is modifying this comment
I think we can also fix this in passing.
======
Kind Regards,
Peter Smith.
Fujitsu Australia.
diff --git a/contrib/test_decoding/expected/binary.out b/contrib/test_decoding/expected/binary.out
index c30abc7..b3a3509 100644
--- a/contrib/test_decoding/expected/binary.out
+++ b/contrib/test_decoding/expected/binary.out
@@ -1,7 +1,11 @@
-- predictability
SET synchronous_commit = on;
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
-ERROR: replication slot "regression_slot" already exists
+ ?column?
+----------
+ init
+(1 row)
+
-- succeeds, textual plugin, textual consumer
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1');
data
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index f763e05..8fd3390 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -103,3 +103,9 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(14 rows)
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 0f62013..91a292b 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,9 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- 'include_generated_columns' option controls whether generated columns
- should be included in the string representation of tuples during
- logical decoding in PostgreSQL. The default is <literal>true</literal>.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>,
</para>
</listitem>
</itemizedlist>
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index bc6033a..25edc6f 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -31,13 +31,11 @@ $node_subscriber->safe_psql('postgres',
# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
-
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
-# publisher-side tab3 has generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
-
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
@@ -60,11 +58,9 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
-
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
);
-
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
);
@@ -98,11 +94,12 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#
+# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is not generated, so confirm that col 'b' IS replicated.
+#
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
-
$node_publisher->wait_for_catchup('sub2');
-
-# the column was NOT replicated (the result value of 'b' is the subscriber-side computed value)
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
@@ -110,10 +107,14 @@ is( $result, qq(4|8
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
+#
+# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
+# can know this because the result value is the subscriber-side computation
+# (which is not the same as the publisher-side computation for col 'b').
+#
$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
-
$node_publisher->wait_for_catchup('sub3');
-
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
is( $result, qq(4|24
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 6e73f89..9804158 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1204,7 +1204,7 @@ t), 'check the number of columns in the old tuple');
# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the columns
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
Attachments:
[text/plain] PS_NITPICKS_20240627_GENCOLS_V110001.txt (5.4K, ../../CAHut+Pv45gB4cV+SSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg@mail.gmail.com/2-PS_NITPICKS_20240627_GENCOLS_V110001.txt)
download | inline diff:
diff --git a/contrib/test_decoding/expected/binary.out b/contrib/test_decoding/expected/binary.out
index c30abc7..b3a3509 100644
--- a/contrib/test_decoding/expected/binary.out
+++ b/contrib/test_decoding/expected/binary.out
@@ -1,7 +1,11 @@
-- predictability
SET synchronous_commit = on;
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
-ERROR: replication slot "regression_slot" already exists
+ ?column?
+----------
+ init
+(1 row)
+
-- succeeds, textual plugin, textual consumer
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1');
data
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index f763e05..8fd3390 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -103,3 +103,9 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(14 rows)
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 0f62013..91a292b 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,9 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- 'include_generated_columns' option controls whether generated columns
- should be included in the string representation of tuples during
- logical decoding in PostgreSQL. The default is <literal>true</literal>.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>,
</para>
</listitem>
</itemizedlist>
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index bc6033a..25edc6f 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -31,13 +31,11 @@ $node_subscriber->safe_psql('postgres',
# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
-
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
-# publisher-side tab3 has generated col 'b' but subscriber-side tab2 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
-
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
@@ -60,11 +58,9 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
-
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
);
-
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
);
@@ -98,11 +94,12 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#
+# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is not generated, so confirm that col 'b' IS replicated.
+#
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
-
$node_publisher->wait_for_catchup('sub2');
-
-# the column was NOT replicated (the result value of 'b' is the subscriber-side computed value)
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
@@ -110,10 +107,14 @@ is( $result, qq(4|8
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
+#
+# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
+# can know this because the result value is the subscriber-side computation
+# (which is not the same as the publisher-side computation for col 'b').
+#
$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
-
$node_publisher->wait_for_catchup('sub3');
-
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
is( $result, qq(4|24
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 6e73f89..9804158 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1204,7 +1204,7 @@ t), 'check the number of columns in the old tuple');
# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the columns
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
[application/octet-stream] PS_NITPICKS_20240627_GENCOLS_V110001_generated_columns.sql (1.2K, ../../CAHut+Pv45gB4cV+SSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg@mail.gmail.com/3-PS_NITPICKS_20240627_GENCOLS_V110001_generated_columns.sql)
download
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-01 10:38 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-07-01 10:38 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Jun 27, 2024 at 2:41 PM Peter Smith <[email protected]> wrote:
>
> Hi, here are some patch v11-0001 comments.
>
> (BTW, I had difficulty reviewing this because something seemed strange
> with the changes this patch made to the test_decoding tests).
>
> ======
> General
>
> 1. Patch name
>
> Patch name does not need to quote 'logical replication'
>
> ~
>
> 2. test_decoding tests
>
> Multiple test_decoding tests were failing for me. There is something
> very suspicious about the unexplained changes the patch made to the
> expected "binary.out" and "decoding_into_rel.out" etc. I REVERTED all
> those changes in my nitpicks top-up to get everything working. Please
> re-confirm that all the test_decoding tests are OK!
>
> ======
> Commit Message
>
> 3.
> Since you are including the example usage for test_decoding, I think
> it's better to include the example usage of CREATE SUBSCRIPTION also.
>
> ======
> contrib/test_decoding/expected/binary.out
>
> 4.
> SELECT 'init' FROM
> pg_create_logical_replication_slot('regression_slot',
> 'test_decoding');
> - ?column?
> -----------
> - init
> -(1 row)
> -
> +ERROR: replication slot "regression_slot" already exists
>
> Huh? Why is this unrelated expected output changed by this patch?
>
> The test_decoding test fails for me unless I REVERT this change!! See
> my nitpicks diff.
>
> ======
> .../expected/decoding_into_rel.out
>
> 5.
> -SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
> - ?column?
> -----------
> - stop
> -(1 row)
> -
>
> Huh? Why is this unrelated expected output changed by this patch?
>
> The test_decoding test fails for me unless I REVERT this change!! See
> my nitpicks diff.
>
> ======
> .../test_decoding/sql/decoding_into_rel.sql
>
> 6.
> -SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
> +SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
>
> Huh, Why does this patch change this code at all? I REVERTED this
> change. See my nitpicks diff.
>
> ======
> .../test_decoding/sql/generated_columns.sql
>
> (see my nitpicks replacement file for this test)
>
> 7.
> +-- test that we can insert the result of a 'include_generated_columns'
> +-- into the tables created. That's really not a good idea in practical terms,
> +-- but provides a nice test.
>
> NITPICK - I didn't understand the point of this comment. I updated
> the comment according to my understanding.
>
> ~
>
> NITPICK - The comment "when 'include-generated-columns' is not set
> then columns will not be replicated" is the opposite of what the
> result is. I changed this comment.
>
> NITPICK - modified and unified wording of some of the other comments
>
> NITPICK - changed some blank lines
>
> ======
> contrib/test_decoding/test_decoding.c
>
> 8.
> + else if (strcmp(elem->defname, "include-generated-columns") == 0)
> + {
> + if (elem->arg == NULL)
> + data->include_generated_columns = true;
>
> Is there any way to test that "elem->arg == NULL" in the
> generated.sql? OTOH, if it is not possible to get here then is the
> code even needed?
>
Currently I could not find a case where the
'include_generated_columns' option is not specifying any value, but I
was hesitant to remove this from here as the other options mentioned
follow the same rules. Thoughts?
> ======
> doc/src/sgml/ddl.sgml
>
> 9.
> <para>
> - Generated columns are skipped for logical replication and cannot be
> - specified in a <command>CREATE PUBLICATION</command> column list.
> + 'include_generated_columns' option controls whether generated columns
> + should be included in the string representation of tuples during
> + logical decoding in PostgreSQL. The default is <literal>true</literal>.
> </para>
>
> NITPICK - Use proper markdown instead of single quotes for the parameter.
>
> NITPICK - I think this can be reworded slightly to provide a
> cross-reference to the CREATE SUBSCRIPTION parameter for more details
> (which means then we can avoid repeating details like the default
> value here). PSA my nitpicks diff for an example of how I thought docs
> should look.
>
> ======
> doc/src/sgml/protocol.sgml
>
> 10.
> + The default is true.
>
> No, it isn't. AFAIK you made the default behaviour true only for
> 'test_decoding', but the default for CREATE SUBSCRIPTION remains
> *false* because that is the existing PG17 behaviour. And the default
> for the 'include_generated_columns' in the protocol is *also* false to
> match the CREATE SUBSCRIPTION default.
>
> e.g. libpqwalreceiver.c only sets ", include_generated_columns 'true'"
> when options->proto.logical.include_generated_columns
> e.g. worker.c says: options->proto.logical.include_generated_columns =
> MySubscription->includegencols;
> e.g. subscriptioncmds.c sets default: opts->include_generated_columns = false;
>
> (This confirmed my previous review expectation that using different
> default behaviours for test_decoding and pgoutput would surely lead to
> confusion)
>
> ~~~
>
> 11.
> - <para>
> - Next, the following message part appears for each column included in
> - the publication (except generated columns):
> - </para>
> -
>
> AFAIK you cannot just remove this entire paragraph because I thought
> it was still relevant to talking about "... the following message
> part". But, if you don't want to explain and cross-reference about
> 'include_generated_columns' then maybe it is OK just to remove the
> "(except generated columns)" part?
>
> ======
> src/test/subscription/t/011_generated.pl
>
> NITPICK - comment typo /tab2/tab3/
> NITPICK - remove some blank lines
>
> ~~~
>
> 12.
> # the column was NOT replicated (the result value of 'b' is the
> subscriber-side computed value)
>
> NITPICK - I think this comment is wrong for the tab2 test because here
> col 'b' IS replicated. I have added much more substantial test case
> comments in the attached nitpicks diff. PSA.
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> 13.
> NITPICK - IMO there is a missing word "list" in the comment. This bug
> existed already on HEAD but since this patch is modifying this comment
> I think we can also fix this in passing.
All the comments are handled.
The attached Patches contain all the suggested changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v13-0001-Enable-support-for-include_generated_columns-opt.patch (88.9K, ../../CAHv8RjLLy=otO_c0y5FN2jipajC2J6+ZRaTcWmEv4X1kmzAjtw@mail.gmail.com/2-v13-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 320397bf8e601cc6694dfe1ab91fae3dd11c1264 Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Mon, 20 May 2024 10:58:31 +0530
Subject: [PATCH v13] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
-- Using Create Subscription
CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=postgres' PUBLICATION pub2 WITH
(include_generated_columns = true, copy_data = false);
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 158 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 6 +
src/test/subscription/t/011_generated.pl | 62 ++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 439 insertions(+), 141 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..9ecd4fa0b7 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..4c3d6ddd12
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated column
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4
+ table public.gencoltable: INSERT: a[integer]:5
+ table public.gencoltable: INSERT: a[integer]:6
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..eab4aa68c9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..9f02f6fbdb
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated column
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index c5e11a6699..a2963054ab 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>,
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..9cf50504a9 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,18 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ The default is false.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6523,11 +6535,6 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</varlistentry>
</variablelist>
- <para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
- </para>
-
<variablelist>
<varlistentry>
<term>Int8</term>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..f942b58565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 02f12f2921..75e7695353 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 95c09c9516..7405eb3deb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..663202832d 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..8fdd1a6591 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5146,6 +5156,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..8c07933d09 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..59f2ce30de 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6598,6 +6598,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..ccff291b85 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* publish generated column data */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..c761c4b829 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..b78e3c6d6a 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +122,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +151,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +163,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +182,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +194,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +229,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +261,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +285,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +338,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +377,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +389,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +402,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +418,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..dbf064474c 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,6 +60,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+-- fail - copy_data and include_generated_columns are mutually exclusive options
+CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
-- fail
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..48efb207e3 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,16 +28,42 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
+
+# publisher-side tab3 has generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+);
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +73,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is($result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is($result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -61,6 +93,34 @@ is( $result, qq(1|22|
3|66|
4|88|
6|132|), 'generated columns replicated');
+#
+# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is not generated, so confirm that col 'b' IS replicated.
+#
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub2');
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+);
+#
+# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
+# can know this because the result value is the subscriber-side computation
+# (which is not the same as the publisher-side computation for col 'b').
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub3');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
# try it with a subscriber-side trigger
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..9804158bb3 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the columns
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
[application/octet-stream] v13-0002-Support-replication-of-generated-column-during-i.patch (20.1K, ../../CAHv8RjLLy=otO_c0y5FN2jipajC2J6+ZRaTcWmEv4X1kmzAjtw@mail.gmail.com/3-v13-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From 9bc60f5f2d1a27031088f25c5d075f2c937996e2 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Tue, 25 Jun 2024 16:32:35 +0530
Subject: [PATCH v13 2/3] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Note that we don't copy columns when the subscriber-side column is also
generated. Those will be filled as normal with the subscriber-side computed or
default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 ---
src/backend/replication/logical/relation.c | 15 ++-
src/backend/replication/logical/tablesync.c | 114 +++++++++++++++-----
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/011_generated.pl | 88 +++++++++++++++
8 files changed, 193 insertions(+), 51 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index f942b58565..408a9157ec 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..27c34059af 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
@@ -427,6 +427,19 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
continue;
}
+ /*
+ * In case 'include_generated_columns' is 'false', we should skip the
+ * check of missing attrs for generated columns.
+ * In case 'include_generated_columns' is 'true', we should check if
+ * corresponding column for the generated column in publication column
+ * list is present in the subscription table.
+ */
+ if (!MySubscription->includegencols && attr->attgenerated)
+ {
+ entry->attrmap->attnums[i] = -1;
+ continue;
+ }
+
attnum = logicalrep_rel_att_by_name(remoterel,
NameStr(attr->attname));
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b00267f042..b3fde6afb3 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -692,20 +693,63 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create list of columns for COPY based on logical relation mapping. Do not
+ * include generated columns of the subscription table in the column list.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *gencollist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ gencollist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
+
+ /* Loop to handle subscription table generated columns. */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if(!remotegenlist[attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'gencollist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ gencollist[attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if(!gencollist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
return attnamelist;
}
@@ -791,16 +835,17 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist_res;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
@@ -948,18 +993,24 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey),"
+ " a.attgenerated != ''"
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
+ walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1024,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist_res = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1057,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist_res[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1069,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist = remotegenlist_res;
walrcv_clear_result(res);
/*
@@ -1123,10 +1177,12 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1191,17 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Regular table with no row filter and 'include_generated_columns'
+ * specified as 'false' during creation of subscription.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL &&
+ !MySubscription->includegencols)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1169,17 +1231,20 @@ copy_table(Relation rel)
else
{
/*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
+ * For non-tables and tables with row filters and when
+ * 'include_generated_columns' is specified as 'true', we need to do
+ * COPY (SELECT ...), as normal COPY of generated column is not
+ * supported. For tables with any row filters, build a SELECT query
+ * with OR'ed row filters for COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_ptr(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1302,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index b78e3c6d6a..d7c4298377 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,9 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index dbf064474c..838881be50 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,9 +60,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 48efb207e3..e225757862 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -16,6 +16,8 @@ $node_publisher->start;
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 10");
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -39,6 +41,28 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+# tab4: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int , b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab5: publisher-side non-generated col 'b' --> subscriber-side generated col 'b'
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# tab6: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# columns on subscriber in different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)");
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab6 (c int GENERATED ALWAYS AS (a * 22) STORED, b int, a int)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -47,6 +71,12 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab5 (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR TABLE tab1");
@@ -54,6 +84,12 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub2 FOR TABLE tab2");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub4 FOR TABLE tab4");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub5 FOR TABLE tab5");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub6 FOR TABLE tab6");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
@@ -65,6 +101,14 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr' PUBLICATION pub4 WITH (include_generated_columns = true)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub6 CONNECTION '$publisher_connstr' PUBLICATION pub6 WITH (include_generated_columns = true)"
+);
+
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -122,6 +166,50 @@ is( $result, qq(4|24
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub4');
+
+# gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
+# gen-col 'c' in publisher not replicating to gen-col 'c' on subscriber
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated column with initial sync');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab6 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub6');
+
+# gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
+# order of column is different on subscriber
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated column with initial sync different column order');
+
+# NOT gen-col 'b' in publisher not replicating to gen-col 'b' on subscriber
+my $offset = -s $node_subscriber->logfile;
+
+# sub5 will cause table sync worker to restart repetitively
+# So SUBSCRIPTION sub5 is created separately
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub5 CONNECTION '$publisher_connstr' PUBLICATION pub5 WITH (include_generated_columns = true)"
+);
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab5" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub5");
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
--
2.34.1
[application/octet-stream] v13-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (15.0K, ../../CAHv8RjLLy=otO_c0y5FN2jipajC2J6+ZRaTcWmEv4X1kmzAjtw@mail.gmail.com/4-v13-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From a6029b0b1a184f88ebb5721a830dc9a46259d4dd Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Fri, 28 Jun 2024 16:42:38 +0530
Subject: [PATCH v13 3/3] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
.../expected/generated_columns.out | 1 +
.../test_decoding/sql/generated_columns.sql | 4 +++-
contrib/test_decoding/test_decoding.c | 8 +++++++-
doc/src/sgml/protocol.sgml | 8 ++++----
doc/src/sgml/ref/create_subscription.sgml | 4 ++--
src/backend/catalog/pg_publication.c | 13 +++++++++++-
src/backend/replication/logical/proto.c | 8 ++++----
src/backend/replication/logical/relation.c | 3 +++
src/backend/replication/logical/tablesync.c | 19 ++++++++++++++----
src/backend/replication/pgoutput/pgoutput.c | 5 ++++-
src/test/subscription/t/011_generated.pl | 20 ++++++++++---------
11 files changed, 66 insertions(+), 27 deletions(-)
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index 4c3d6ddd12..95d313d3b8 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -50,3 +50,4 @@ SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
stop
(1 row)
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 9f02f6fbdb..c378c1f0a9 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -19,4 +19,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..7aca5a19ac 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,7 +557,13 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
+ /*
+ * Don't print virtual generated column. Don't print stored
+ * generated column if 'include_generated_columns' is false.
+ *
+ * TODO: can use ATTRIBUTE_GENERATED_VIRTUAL to simpilfy
+ */
+ if (attr->attgenerated && (attr->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
/*
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 39207a6755..dd03aab60b 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3310,10 +3310,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
- The default is true.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL. The default is true.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..e5e5aef243 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -533,6 +534,16 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ /*
+ * TODO: simplify the expression
+ */
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated &&
+ TupleDescAttr(tupdesc, attnum - 1)->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot use virtual generated column \"%s\" in publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1225,7 +1236,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped)
+ if (att->attisdropped || (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED))
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 7405eb3deb..e82e53e384 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,7 +793,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -817,7 +817,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -957,7 +957,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -981,7 +981,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 27c34059af..e1b1693700 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -427,6 +427,9 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
continue;
}
+ if (attr->attgenerated && attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
/*
* In case 'include_generated_columns' is 'false', we should skip the
* check of missing attrs for generated columns.
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b3fde6afb3..d44f10901e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -712,7 +712,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (!attr->attgenerated)
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
attnum = logicalrep_rel_att_by_name(&rel->remoterel,
@@ -1001,10 +1001,21 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
" WHERE a.attnum > 0::pg_catalog.int2"
" AND NOT a.attisdropped", lrel->remoteid);
- if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
- walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
- !MySubscription->includegencols)
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000)
+ {
+ bool gencols_allowed = walrcv_server_version(LogRepWorkerWalRcvConn) >= 170000
+ && MySubscription->includegencols;
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
+ {
+ /* Replication of generated cols is not supported. */
appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+ }
appendStringInfo(&cmd,
" AND a.attrelid = %u"
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 00c6566959..69aaf849e4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -784,7 +784,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -1106,6 +1106,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index e225757862..dd27a6e34b 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -30,18 +30,18 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+# publisher-side tab2 has stored generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
-# publisher-side tab3 has generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has stored generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
-# tab4: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# tab4: publisher-side stored generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int , b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -50,19 +50,21 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
);
-# tab5: publisher-side non-generated col 'b' --> subscriber-side generated col 'b'
+# tab5: publisher-side non-generated col 'b' --> subscriber-side stored generated col 'b'
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
-# tab6: publisher-side generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and generated-col 'c'
+# tab6: publisher-side stored generated col 'b' and 'c' --> subscriber-side non-generated col 'b', and stored generated-col 'c'
# columns on subscriber in different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab6 (c int GENERATED ALWAYS AS (a * 22) STORED, b int, a int)");
+# TODO: Add tests related to replication of VIRTUAL GNERATED COLUMNS
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -170,8 +172,8 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub4');
-# gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
-# gen-col 'c' in publisher not replicating to gen-col 'c' on subscriber
+# stored gen-col 'b' in publisher replicating to NOT gen-col 'b' on subscriber
+# stored gen-col 'c' in publisher not replicating to stored gen-col 'c' on subscriber
$result =
$node_subscriber->safe_psql('postgres',
"SELECT a, b, c FROM tab4 ORDER BY a");
@@ -185,7 +187,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab6 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub6');
-# gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
+# stored gen-col 'b' and 'c' in publisher replicating to NOT gen-col 'b' and gen-col 'c' on subscriber
# order of column is different on subscriber
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b, c FROM tab6 ORDER BY a");
@@ -195,7 +197,7 @@ is( $result, qq(1|2|22
4|8|88
5|10|110), 'replicate generated column with initial sync different column order');
-# NOT gen-col 'b' in publisher not replicating to gen-col 'b' on subscriber
+# NOT gen-col 'b' in publisher not replicating to stored gen-col 'b' on subscriber
my $offset = -s $node_subscriber->logfile;
# sub5 will cause table sync worker to restart repetitively
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-02 04:23 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-02 04:23 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Jul 1, 2024 at 8:38 PM Shubham Khanna
<[email protected]> wrote:
>
>...
> > 8.
> > + else if (strcmp(elem->defname, "include-generated-columns") == 0)
> > + {
> > + if (elem->arg == NULL)
> > + data->include_generated_columns = true;
> >
> > Is there any way to test that "elem->arg == NULL" in the
> > generated.sql? OTOH, if it is not possible to get here then is the
> > code even needed?
> >
>
> Currently I could not find a case where the
> 'include_generated_columns' option is not specifying any value, but I
> was hesitant to remove this from here as the other options mentioned
> follow the same rules. Thoughts?
>
If you do manage to find a scenario for this then I think a test for
it would be good. But, I agree that the code seems OK because now I
see it is the same pattern as similar nearby code.
~~~
Thanks for the updated patch. Here are some review comments for patch v13-0001.
======
.../expected/generated_columns.out
nitpicks (see generated_columns.sql)
======
.../test_decoding/sql/generated_columns.sql
nitpick - use plural /column/columns/
nitpick - use consistent wording in the comments
nitpick - IMO it is better to INSERT different values for each of the tests
======
doc/src/sgml/protocol.sgml
nitpick - I noticed that none of the other boolean parameters on this
page mention about a default, so maybe here we should do the same and
omit that information.
~~~
1.
- <para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
- </para>
-
In a previous review [1 comment #11] I wrote that you can't just
remove this paragraph because AFAIK it is still meaningful. A minimal
change might be to just remove the "(except generated columns)" part.
Alternatively, you could give a more detailed explanation mentioning
the include_generated_columns protocol parameter.
I provided some updated text for this paragraph in my NITPICKS top-up
patch, Please have a look at that for ideas.
======
src/backend/commands/subscriptioncmds.c
It looks like pg_indent needs to be run on this file.
======
src/include/catalog/pg_subscription.h
nitpick - comment /publish/Publish/ for consistency
======
src/include/replication/walreceiver.h
nitpick - comment /publish/Publish/ for consistency
======
src/test/regress/expected/subscription.out
nitpicks - (see subscription.sql)
======
src/test/regress/sql/subscription.sql
nitpick - combine the invalid option combinations test with all the
others (no special comment needed)
nitpick - rename subscription as 'regress_testsub2' same as all its peers.
======
src/test/subscription/t/011_generated.pl
nitpick - add/remove blank lines
======
src/test/subscription/t/031_column_list.pl
nitpick - rewording for a comment. This issue was not strictly caused
by this patch, but since you are modifying the same comment we can fix
this in passing.
======
99.
Please also see the attached top-up patch for all those nitpicks
identified above.
======
[1] v11-0001 review
https://www.postgresql.org/message-id/CAHut%2BPv45gB4cV%2BSSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg%40mail...
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index 4c3d6dd..f3b26aa 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -1,4 +1,4 @@
--- test decoding of generated column
+-- test decoding of generated columns
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
?column?
----------
@@ -7,7 +7,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d
-- column b' is a generated column
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- when 'include-generated-columns' is not set the generated column 'b' will be replicated
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
INSERT INTO gencoltable (a) VALUES (1), (2), (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
data
@@ -20,26 +20,26 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
(5 rows)
-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
-INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
- data
--------------------------------------------------------------
+ data
+--------------------------------------------------------------
BEGIN
- table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
- table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
- table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
COMMIT
(5 rows)
-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
-INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
data
------------------------------------------------
BEGIN
- table public.gencoltable: INSERT: a[integer]:4
- table public.gencoltable: INSERT: a[integer]:5
- table public.gencoltable: INSERT: a[integer]:6
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
COMMIT
(5 rows)
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 9f02f6f..a5bb598 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -1,22 +1,22 @@
--- test decoding of generated column
+-- test decoding of generated columns
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
-- column b' is a generated column
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- when 'include-generated-columns' is not set the generated column 'b' will be replicated
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
INSERT INTO gencoltable (a) VALUES (1), (2), (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
-INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
-INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9cf5050..226c364 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3313,7 +3313,6 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
Boolean option to enable generated columns. This option controls
whether generated columns should be included in the string
representation of tuples during logical decoding in PostgreSQL.
- The default is false.
</para>
</listitem>
</varlistentry>
@@ -6535,6 +6534,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</varlistentry>
</variablelist>
+ <para>
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
+ </para>
+
<variablelist>
<varlistentry>
<term>Int8</term>
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index ccff291..4663f7c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -160,7 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
- bool includegencols; /* publish generated column data */
+ bool includegencols; /* Publish generated column data */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c761c4b..9275b3a 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,7 +186,7 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
- bool include_generated_columns; /* publish generated
+ bool include_generated_columns; /* Publish generated
* columns */
} logical;
} proto;
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index b78e3c6..e8824fa 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,11 +99,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
-CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index dbf0644..8c63c13 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,12 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-- fail - include_generated_columns must be boolean
-CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 48efb20..25edc6f 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -93,20 +93,20 @@ is( $result, qq(1|22|
3|66|
4|88|
6|132|), 'generated columns replicated');
+
#
# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
# col 'b' is not generated, so confirm that col 'b' IS replicated.
#
-
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
-
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
5|10),
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
+
#
# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9804158..5bfed27 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1204,7 +1204,7 @@ t), 'check the number of columns in the old tuple');
# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column list (aka all columns as part of the columns
+# publication without any column list (aka all columns are part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
Attachments:
[text/plain] PS_NITPICKS_20240702_v130001.txt (11.8K, ../../CAHut+PvST7be97xnxFjiHpz=WSr9MV567DLP4O-o7=Kb26pHTA@mail.gmail.com/2-PS_NITPICKS_20240702_v130001.txt)
download | inline diff:
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index 4c3d6dd..f3b26aa 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -1,4 +1,4 @@
--- test decoding of generated column
+-- test decoding of generated columns
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
?column?
----------
@@ -7,7 +7,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d
-- column b' is a generated column
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- when 'include-generated-columns' is not set the generated column 'b' will be replicated
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
INSERT INTO gencoltable (a) VALUES (1), (2), (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
data
@@ -20,26 +20,26 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
(5 rows)
-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
-INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
- data
--------------------------------------------------------------
+ data
+--------------------------------------------------------------
BEGIN
- table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
- table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
- table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
COMMIT
(5 rows)
-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
-INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
data
------------------------------------------------
BEGIN
- table public.gencoltable: INSERT: a[integer]:4
- table public.gencoltable: INSERT: a[integer]:5
- table public.gencoltable: INSERT: a[integer]:6
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
COMMIT
(5 rows)
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 9f02f6f..a5bb598 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -1,22 +1,22 @@
--- test decoding of generated column
+-- test decoding of generated columns
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
-- column b' is a generated column
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- when 'include-generated-columns' is not set the generated column 'b' will be replicated
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
INSERT INTO gencoltable (a) VALUES (1), (2), (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
-INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
-INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9cf5050..226c364 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3313,7 +3313,6 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
Boolean option to enable generated columns. This option controls
whether generated columns should be included in the string
representation of tuples during logical decoding in PostgreSQL.
- The default is false.
</para>
</listitem>
</varlistentry>
@@ -6535,6 +6534,13 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</varlistentry>
</variablelist>
+ <para>
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
+ </para>
+
<variablelist>
<varlistentry>
<term>Int8</term>
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index ccff291..4663f7c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -160,7 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
- bool includegencols; /* publish generated column data */
+ bool includegencols; /* Publish generated column data */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c761c4b..9275b3a 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,7 +186,7 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
- bool include_generated_columns; /* publish generated
+ bool include_generated_columns; /* Publish generated
* columns */
} logical;
} proto;
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index b78e3c6..e8824fa 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,11 +99,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
-CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index dbf0644..8c63c13 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,12 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-
--- fail - copy_data and include_generated_columns are mutually exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-- fail - include_generated_columns must be boolean
-CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 48efb20..25edc6f 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -93,20 +93,20 @@ is( $result, qq(1|22|
3|66|
4|88|
6|132|), 'generated columns replicated');
+
#
# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
# col 'b' is not generated, so confirm that col 'b' IS replicated.
#
-
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
-
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
5|10),
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
+
#
# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9804158..5bfed27 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1204,7 +1204,7 @@ t), 'check the number of columns in the old tuple');
# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column list (aka all columns as part of the columns
+# publication without any column list (aka all columns are part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-02 05:29 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-02 05:29 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham,
As you can see, most of my recent review comments for patch 0001 are
only cosmetic nitpicks. But, there is still one long-unanswered design
question from a month ago [1, #G.2]
A lot of the patch code of pgoutput.c and proto.c and logicalproto.h
is related to the introduction and passing everywhere of new
'include_generated_columns' function parameters. These same functions
are also always passing "BitMapSet *columns" representing the
publication column list.
My question was about whether we can't make use of the existing BMS
parameter instead of introducing all the new API parameters.
The idea might go something like this:
* If 'include_generated_columns' option is specified true and if no
column list was already specified then perhaps the relentry->columns
can be used for a "dummy" column list that has everything including
all the generated columns.
* By doing this:
-- you may be able to avoid passing the extra
'include_gernated_columns' everywhere
-- you may be able to avoid checking for generated columns deeper in
the code (since it is already checked up-front when building the
column list BMS)
~~
I'm not saying this design idea is guaranteed to work, but it might be
worth considering, because if it does work then there is potential to
make the current 0001 patch significantly shorter.
======
[1] https://www.postgresql.org/message-id/CAHut%2BPsuJfcaeg6zst%3D6PE5uyJv_UxVRHU3ck7W2aHb1uQYKng%40mail...
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-05 12:01 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-07-05 12:01 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Tue, Jul 2, 2024 at 9:53 AM Peter Smith <[email protected]> wrote:
>
> On Mon, Jul 1, 2024 at 8:38 PM Shubham Khanna
> <[email protected]> wrote:
> >
> >...
> > > 8.
> > > + else if (strcmp(elem->defname, "include-generated-columns") == 0)
> > > + {
> > > + if (elem->arg == NULL)
> > > + data->include_generated_columns = true;
> > >
> > > Is there any way to test that "elem->arg == NULL" in the
> > > generated.sql? OTOH, if it is not possible to get here then is the
> > > code even needed?
> > >
> >
> > Currently I could not find a case where the
> > 'include_generated_columns' option is not specifying any value, but I
> > was hesitant to remove this from here as the other options mentioned
> > follow the same rules. Thoughts?
> >
>
> If you do manage to find a scenario for this then I think a test for
> it would be good. But, I agree that the code seems OK because now I
> see it is the same pattern as similar nearby code.
>
> ~~~
>
> Thanks for the updated patch. Here are some review comments for patch v13-0001.
>
> ======
> .../expected/generated_columns.out
>
> nitpicks (see generated_columns.sql)
>
> ======
> .../test_decoding/sql/generated_columns.sql
>
> nitpick - use plural /column/columns/
> nitpick - use consistent wording in the comments
> nitpick - IMO it is better to INSERT different values for each of the tests
>
> ======
> doc/src/sgml/protocol.sgml
>
> nitpick - I noticed that none of the other boolean parameters on this
> page mention about a default, so maybe here we should do the same and
> omit that information.
>
> ~~~
>
> 1.
> - <para>
> - Next, the following message part appears for each column included in
> - the publication (except generated columns):
> - </para>
> -
>
> In a previous review [1 comment #11] I wrote that you can't just
> remove this paragraph because AFAIK it is still meaningful. A minimal
> change might be to just remove the "(except generated columns)" part.
> Alternatively, you could give a more detailed explanation mentioning
> the include_generated_columns protocol parameter.
>
> I provided some updated text for this paragraph in my NITPICKS top-up
> patch, Please have a look at that for ideas.
>
> ======
> src/backend/commands/subscriptioncmds.c
>
> It looks like pg_indent needs to be run on this file.
>
> ======
> src/include/catalog/pg_subscription.h
>
> nitpick - comment /publish/Publish/ for consistency
>
> ======
> src/include/replication/walreceiver.h
>
> nitpick - comment /publish/Publish/ for consistency
>
> ======
> src/test/regress/expected/subscription.out
>
> nitpicks - (see subscription.sql)
>
> ======
> src/test/regress/sql/subscription.sql
>
> nitpick - combine the invalid option combinations test with all the
> others (no special comment needed)
> nitpick - rename subscription as 'regress_testsub2' same as all its peers.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> nitpick - add/remove blank lines
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> nitpick - rewording for a comment. This issue was not strictly caused
> by this patch, but since you are modifying the same comment we can fix
> this in passing.
>
> ======
> 99.
> Please also see the attached top-up patch for all those nitpicks
> identified above.
>
> ======
> [1] v11-0001 review
> https://www.postgresql.org/message-id/CAHut%2BPv45gB4cV%2BSSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg%40mail...
All the comments are handled.
The attached Patches contain all the suggested changes. Here, v15-0001
is modified to fix the comments, v15-0002 is not modified and v15-0003
is modified according to the changes in v15-0001 patch.
Thanks Shlok Kyal for modifying the v15-0003 Patch.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v15-0002-Support-replication-of-generated-column-during-i.patch (25.9K, ../../CAHv8Rj+=hn--ALJQvzzu7meX3LuO3tJKppDS7eO1BGvNFYBAbg@mail.gmail.com/2-v15-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From bcbca02941230fa830926ee4ddfd856c7863f864 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Fri, 5 Jul 2024 15:37:44 +0530
Subject: [PATCH v15 2/3] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Summary:
when (include_generated_column = true)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column: The
publisher generated column value is copied.
* publisher generated column => subscriber generated column: The
publisher generated column value is not copied. The subscriber
generated column will be filled with the subscriber-side computed or
default data.
when (include_generated_columns = false)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column: This
will replicate nothing. Publisher generate-column is not replicated.
The subscriber column will be filled with the subscriber-side default
data.
* publisher generated column => subscriber generated column: This
will replicate nothing. Publisher generate-column is not replicated.
The subscriber generated column will be filed with the
subscriber-side computed or default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 --
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/tablesync.c | 134 ++++++++++++++++----
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/004_sync.pl | 42 ++++++
src/test/subscription/t/011_generated.pl | 124 +++++++++++++++++-
9 files changed, 274 insertions(+), 55 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index f942b58565..408a9157ec 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..9de0b75330 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b00267f042..38f3621c85 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -692,21 +693,68 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create list of columns for COPY based on logical relation mapping. Do not
+ * include generated columns of the subscription table in the column list.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *localgenlist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ localgenlist = palloc0((rel->remoterel.natts * sizeof(bool)));
+
+ /*
+ * This loop checks for generated columns on subscription table.
+ */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if (!remotegenlist[attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'localgenlist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ localgenlist[attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY, excluding columns that are
+ * subscription table generated columns.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if (!localgenlist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
+ pfree(localgenlist);
return attnamelist;
}
@@ -791,19 +839,21 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
+ int server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
lrel->nspname = nspname;
lrel->relname = relname;
@@ -851,7 +901,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
WalRcvExecResult *pubres;
TupleTableSlot *tslot;
@@ -948,18 +998,27 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey)");
+
+ if(server_version >= 120000)
+ appendStringInfo(&cmd, ", a.attgenerated != ''");
+
+ appendStringInfo(&cmd,
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if ((server_version >= 120000 && server_version <= 160000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1032,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1065,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1077,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist_res = remotegenlist;
walrcv_clear_result(res);
/*
@@ -1037,7 +1099,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* 3) one of the subscribed publications is declared as TABLES IN SCHEMA
* that includes this relation
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
StringInfoData pub_names;
@@ -1123,10 +1185,13 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
+ bool remote_has_gencol = false;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1200,29 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /* check if remote column list has generated columns */
+ if(MySubscription->includegencols)
+ {
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if(remotegenlist[i])
+ {
+ remote_has_gencol = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * Regular table with no row filter and 'include_generated_columns'
+ * specified as 'false' during creation of subscription.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL && !remote_has_gencol)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1173,13 +1256,19 @@ copy_table(Relation rel)
* (SELECT ...), but we can't just do SELECT * because we need to not
* copy generated columns. For tables with any row filters, build a
* SELECT query with OR'ed row filters for COPY.
+ *
+ * We also need to use this same COPY (SELECT ...) syntax when
+ * 'include_generated_columns' is specified as true, because copy
+ * of generated columns is not supported by the normal COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_node(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1326,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 36916c0ac2..592c1f0667 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,9 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - include_generated_columns and copy_data = true are mutually exclusive
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7944152124..8c7381fbfc 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,9 +60,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
--- fail - include_generated_columns and copy_data = true are mutually exclusive
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
-
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index a2d9462395..68052e144e 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -172,6 +172,48 @@ ok( $node_publisher->poll_query_until(
'postgres', 'SELECT count(*) = 0 FROM pg_replication_slots'),
'DROP SUBSCRIPTION during error can clean up the slots on the publisher');
+# clean up
+$node_publisher->safe_psql('postgres', "DROP TABLE tab_rep");
+$node_subscriber->safe_psql('postgres', "DROP TABLE tab_rep");
+
+# When a subscription table have a column missing as specified on publication table
+# setup structure with existing data on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_rep (a int, b int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_rep VALUES (1, 1), (2, 2), (3, 3)");
+
+# add table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_rep (a int)");
+
+my $offset = -s $node_subscriber->logfile;
+
+# recreate the subscription again
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+);
+
+# check for missing column error
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_rep" is missing replicated column: "b"/,
+ $offset);
+
+# clean up
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub");
+$node_subscriber->safe_psql('postgres', "DROP TABLE tab_rep");
+
+# When a subscription table have a generated column corresponding to non-generated column on publication table
+# create table on subscriber side with generated column
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_rep (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+);
+
+# check for missing column error
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_rep" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 25edc6fa17..0b350e388d 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -16,6 +16,8 @@ $node_publisher->start;
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 10");
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -39,6 +41,31 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+# publisher-side tab4 has generated cols 'b' and 'c' but subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
+# where columns on the subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab4 (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# publisher-side tab5 has non-generated col 'b' but subscriber-side tab5 has generated col 'b'
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# test for alter subscription ... refresh publication
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -47,6 +74,12 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab5 (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR TABLE tab1");
@@ -54,15 +87,22 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub2 FOR TABLE tab2");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub4 FOR TABLE tab4");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub5 FOR TABLE tab5");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr' PUBLICATION pub4 WITH (include_generated_columns = true)"
);
# Wait for initial sync of all subscriptions
@@ -74,10 +114,20 @@ is( $result, qq(1|22
3|66), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'generated column initial sync');
# data to replicate
@@ -102,7 +152,10 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
@@ -117,11 +170,70 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub3');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
-is( $result, qq(4|24
+is( $result, qq(1|21
+2|22
+3|23
+4|24
5|25),
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
+# TEST tab4: the publisher-side cols 'b' and 'c' are generated and subscriber-side
+# col 'b' is not generated and col 'c' is generated. So confirmed that the different
+# order of columns on subsriber-side replicate data to correct columns.
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub4');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated columns with different order on subscriber');
+
+# TEST for ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+# Add a new table to publication
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub4 ADD TABLE tab6");
+
+# Refresh publication after table is added to publication
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub4 REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('sub4');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'add new table to existing publication');
+
+# TEST: drop generated column on subscriber side
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (4), (5)");
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|8
+5|10|10), 'add new table to existing publication');
+
+# TEST tab5: publisher-side col 'b' is not-generated and subscriber-side col 'b' is generated.
+# so confirmed that col 'b' IS NOT replicated and it will throw an error.
+# SUBSCRIPTION sub5 is created separately as sub5 will cause table sync worker to restart
+# repetitively
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub5 CONNECTION '$publisher_connstr' PUBLICATION pub5 WITH (include_generated_columns = true)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab5" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub5");
+
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
--
2.34.1
[application/octet-stream] v15-0001-Enable-support-for-include_generated_columns-opt.patch (83.3K, ../../CAHv8Rj+=hn--ALJQvzzu7meX3LuO3tJKppDS7eO1BGvNFYBAbg@mail.gmail.com/3-v15-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From f5c808a23febed8c87df4595015f5978e4edeec6 Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Mon, 20 May 2024 10:58:31 +0530
Subject: [PATCH v15 1/3] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
-- Using Create Subscription
CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=postgres' PUBLICATION pub2 WITH
(include_generated_columns = true, copy_data = false);
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 8 +-
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 93 ++++++++++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 3 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 158 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 6 +
src/test/subscription/t/011_generated.pl | 62 ++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 457 insertions(+), 117 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..9ecd4fa0b7 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..eab4aa68c9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index c5e11a6699..a2963054ab 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>,
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..226c3641b9 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6524,8 +6535,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..f942b58565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 02f12f2921..75e7695353 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 95c09c9516..24528dc41e 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -781,7 +781,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +802,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -938,7 +938,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +959,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3b285894db..3e6d68a3d6 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..5ff5078bbc 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -164,8 +164,10 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns included in the publication, or NULL if all columns are
- * included implicitly. Note that the attnums in this bitmap are not
+ * Columns should be publicated, or NULL if all columns are included
+ * implicitly. This bitmap only considers the column list of the
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
@@ -283,11 +285,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +400,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -766,7 +780,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -1008,6 +1022,34 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Prepare new column list bitmap. This includes all the columns of the table.
+ */
+static Bitmapset *
+prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
+ TupleDesc desc)
+{
+ Bitmapset *cols = NULL;
+ MemoryContext oldcxt = NULL;
+
+ pgoutput_ensure_entry_cxt(data, entry);
+ oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped)
+ continue;
+
+ cols = bms_add_member(cols, i + 1);
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+
+ return cols;
+}
+
/*
* Initialize the column list.
*/
@@ -1085,7 +1127,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1095,7 +1137,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
* If column list includes all the columns of the table,
* set it to NULL.
*/
- if (bms_num_members(cols) == nliveatts)
+ if (bms_num_members(cols) == nliveatts &&
+ data->include_generated_columns)
{
bms_free(cols);
cols = NULL;
@@ -1106,6 +1149,46 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
}
}
+ /* Do additional checks if the generated columns must be replicated */
+ if (!data->include_generated_columns)
+ {
+ TupleDesc desc = RelationGetDescr(relation);
+ int nliveatts = 0;
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped */
+ if (att->attisdropped)
+ continue;
+
+ /* Count all valid attributes */
+ nliveatts++;
+
+ /* Skip if the attribute is not generated */
+ if (!att->attgenerated)
+ continue;
+
+ /* Prepare new bms if not allocated yet */
+ if (cols == NULL)
+ cols = prepare_all_columns_bms(data, entry, desc);
+
+ /* Delete the corresponding column from the bms */
+ cols = bms_del_member(cols, i + 1);
+ }
+
+ /*
+ * If column list includes all the columns of the table, set it to
+ * NULL.
+ */
+ if (bms_num_members(cols) == nliveatts)
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
+ }
+
if (first)
{
entry->columns = cols;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5426f1177c..9459138bbf 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4754,6 +4754,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4826,11 +4827,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4869,6 +4876,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4915,6 +4923,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..f7b8d59413 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..432a164e18 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..0bb578221b 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns data */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..b9a64d9c95 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -230,7 +230,8 @@ extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..9275b3a617 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..36916c0ac2 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+-- fail - include_generated_columns and copy_data = true are mutually exclusive
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +122,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +151,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +163,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +182,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +194,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +229,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +261,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +285,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +338,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +377,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +389,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +402,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +418,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..7944152124 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -60,6 +60,12 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+-- fail - include_generated_columns and copy_data = true are mutually exclusive
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
-- fail
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..25edc6fa17 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,16 +28,42 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
+
+# publisher-side tab3 has generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+);
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +73,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is($result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is($result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,6 +94,34 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#
+# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is not generated, so confirm that col 'b' IS replicated.
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub2');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+);
+
+#
+# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
+# can know this because the result value is the subscriber-side computation
+# (which is not the same as the publisher-side computation for col 'b').
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub3');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.34.1
[application/octet-stream] v15-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (12.7K, ../../CAHv8Rj+=hn--ALJQvzzu7meX3LuO3tJKppDS7eO1BGvNFYBAbg@mail.gmail.com/4-v15-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From beb3c655866d9b2064458b93a4f9e95a84bc0985 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Fri, 5 Jul 2024 15:29:00 +0530
Subject: [PATCH v15 3/3] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
.../expected/generated_columns.out | 1 +
.../test_decoding/sql/generated_columns.sql | 4 +++-
contrib/test_decoding/test_decoding.c | 15 +++++++++++++--
doc/src/sgml/protocol.sgml | 7 ++++---
doc/src/sgml/ref/create_subscription.sgml | 4 ++--
src/backend/catalog/pg_publication.c | 12 ++++++++++++
src/backend/replication/logical/proto.c | 12 ++++++++++++
src/backend/replication/logical/tablesync.c | 19 ++++++++++++++++---
src/backend/replication/pgoutput/pgoutput.c | 12 ++++++++++++
src/test/subscription/t/011_generated.pl | 8 ++++----
10 files changed, 79 insertions(+), 15 deletions(-)
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index f3b26aa9e1..a79510705c 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -50,3 +50,4 @@ SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
stop
(1 row)
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 6d6d1d6564..997cdebc7e 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -19,4 +19,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..a847050f6e 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,8 +557,19 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
- continue;
+ if (attr->attgenerated)
+ {
+ /*
+ * Don't print generated columns when
+ * 'include_generated_columns' is false.
+ */
+ if (!include_generated_columns)
+ continue;
+
+ /* Don't print generated columns unless they are STORED. */
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
/*
* Don't print system columns, oid will already have been printed if
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 226c3641b9..06554fb2af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3310,9 +3310,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..1809e140ea 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -520,6 +521,7 @@ publication_translate_columns(Relation targetrel, List *columns,
{
char *colname = strVal(lfirst(lc));
AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+ Form_pg_attribute att;
if (attnum == InvalidAttrNumber)
ereport(ERROR,
@@ -533,6 +535,13 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ att = TupleDescAttr(tupdesc, attnum - 1);
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("generated column \"%s\" is not STORED so cannot be used in a publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1228,6 +1237,9 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
attnums[nattnums++] = att->attnum;
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 24528dc41e..789914a4b5 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -784,6 +784,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -805,6 +808,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -941,6 +947,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -962,6 +971,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 38f3621c85..ad1a83d169 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -714,7 +714,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (!attr->attgenerated)
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
attnum = logicalrep_rel_att_by_name(&rel->remoterel,
@@ -1010,9 +1010,22 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
" WHERE a.attnum > 0::pg_catalog.int2"
" AND NOT a.attisdropped", lrel->remoteid);
- if ((server_version >= 120000 && server_version <= 160000) ||
- !MySubscription->includegencols)
+ if(server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 170000 && MySubscription->includegencols;
+
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ /* TODO: use ATTRIBUTE_GENERATED_VIRTUAL*/
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
+ {
+ /* Replication of generated cols is not supported. */
appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+ }
appendStringInfo(&cmd,
" AND a.attrelid = %u"
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5ff5078bbc..b7fd4c3882 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -783,6 +783,9 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -1042,6 +1045,9 @@ prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
cols = bms_add_member(cols, i + 1);
}
@@ -1130,6 +1136,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
@@ -1163,6 +1172,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
/* Count all valid attributes */
nliveatts++;
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 0b350e388d..1f8a59d8d9 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -30,18 +30,18 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+# publisher-side tab2 has stored generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
-# publisher-side tab3 has generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has stored generated col 'b' but subscriber-side tab3 has DIFFERENT COMPUTATION stored generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
-# publisher-side tab4 has generated cols 'b' and 'c' but subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
+# publisher-side tab4 has stored generated cols 'b' and 'c' but subscriber-side tab4 has non-generated col 'b', and stored generated-col 'c'
# where columns on the subscriber are in a different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -51,7 +51,7 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab4 (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
);
-# publisher-side tab5 has non-generated col 'b' but subscriber-side tab5 has generated col 'b'
+# publisher-side tab5 has non-generated col 'b' but subscriber-side tab5 has stored generated col 'b'
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-05 12:05 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-07-05 12:05 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Tue, Jul 2, 2024 at 10:59 AM Peter Smith <[email protected]> wrote:
>
> Hi Shubham,
>
> As you can see, most of my recent review comments for patch 0001 are
> only cosmetic nitpicks. But, there is still one long-unanswered design
> question from a month ago [1, #G.2]
>
> A lot of the patch code of pgoutput.c and proto.c and logicalproto.h
> is related to the introduction and passing everywhere of new
> 'include_generated_columns' function parameters. These same functions
> are also always passing "BitMapSet *columns" representing the
> publication column list.
>
> My question was about whether we can't make use of the existing BMS
> parameter instead of introducing all the new API parameters.
>
> The idea might go something like this:
>
> * If 'include_generated_columns' option is specified true and if no
> column list was already specified then perhaps the relentry->columns
> can be used for a "dummy" column list that has everything including
> all the generated columns.
>
> * By doing this:
> -- you may be able to avoid passing the extra
> 'include_gernated_columns' everywhere
> -- you may be able to avoid checking for generated columns deeper in
> the code (since it is already checked up-front when building the
> column list BMS)
>
> ~~
>
> I'm not saying this design idea is guaranteed to work, but it might be
> worth considering, because if it does work then there is potential to
> make the current 0001 patch significantly shorter.
>
> ======
> [1] https://www.postgresql.org/message-id/CAHut%2BPsuJfcaeg6zst%3D6PE5uyJv_UxVRHU3ck7W2aHb1uQYKng%40mail...
I have fixed this issue in the latest Patches.
Please refer to the updated v15 Patches here in [1]. See [1] for the
changes added.
[1] https://www.postgresql.org/message-id/CAHv8Rj%2B%3Dhn--ALJQvzzu7meX3LuO3tJKppDS7eO1BGvNFYBAbg%40mail...
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-08 05:22 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-08 05:22 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Here are review comments for v15-0001
======
doc/src/sgml/ddl.sgml
nitpick - there was a comma (,) which should be a period (.)
======
.../libpqwalreceiver/libpqwalreceiver.c
1.
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
Should now say >= 180000
======
src/backend/replication/pgoutput/pgoutput.c
nitpick - comment wording for RelationSyncEntry.collist.
~~
2.
pgoutput_column_list_init:
I found the current logic to be quite confusing. I assume the code is
working OK, because AFAIK there are plenty of tests and they are all
passing, but the logic seems somewhat repetitive and there are also no
comments to explain it adding to my confusion.
IIUC, PRIOR TO THIS PATCH:
BMS field 'columns' represented the "columns of the column list" or it
was NULL if there was no publication column list (and it was also NULL
if the column list contained every column).
IIUC NOW, WITH THIS PATCH:
The BMS field 'columns' meaning is changed slightly to be something
like "columns to be replicated" or NULL if all columns are to be
replicated. This is almost the same thing except we are now handing
the generated columns up-front, so generated columns will or won't
appear in the BMS according to the "include_generated_columns"
parameter. See how this is all a bit subtle which is why copious new
comments are required to explain it...
So, although the test result evidence suggests this is working OK, I
have many questions/issues about it. Here are some to start with:
2a. It needs a lot more (summary and detailed) comments explaining the
logic now that the meaning is slightly different.
2b. What is the story with the FOR ALL TABLES case now? Previously,
there would always be NULL 'columns' for "FOR ALL TABLES" case -- the
comment still says so. But now you've tacked on a 2nd pass of
iterations to build the BMS outside of the "if (!pub->alltables)"
check. Is that OK?
2c. The following logic seemed unexpected:
- if (bms_num_members(cols) == nliveatts)
+ if (bms_num_members(cols) == nliveatts &&
+ data->include_generated_columns)
{
bms_free(cols);
cols = NULL;
`
I had thought the above code would look different -- more like:
if (att->attgenerated && !data->include_generated_columns)
continue;
nliveatts++;
...
2d. Was so much duplicated code necessary? It feels like the whole
"Get the number of live attributes." and assignment of cols to NULL
might be made common to both code paths.
2e. I'm beginning to question the pros/cons of the new BMS logic; I
had suggested trying this way (processing the generated columns
up-front in the BMS 'columns' list) to reduce patch code and simplify
all the subsequent API delegation of "include_generated_cloumns"
everywhere like it was in v14-0001. Indeed, that part was a success
and the patch is now smaller. But I don't like much that we've traded
reduced code overall for increased confusing code in that BMS
function. If all this BMS code can be refactored and commented to be
easier to understand then maybe all will be well, but if it can't then
maybe this BMS change was a bridge too far. I haven't given up on it
just yet, but I wonder what was your opinion about it, and do other
people have thoughts about whether this was the good direction to
take?
======
src/bin/pg_dump/pg_dump.c
3.
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
Should now say >= 180000
======
src/bin/psql/describe.c
4.
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
Should now say >= 180000
======
src/include/catalog/pg_subscription.h
nitpick - let's make the comment the same as in WalRcvStreamOptions
======
src/include/replication/logicalproto.h
nitpick - extern for logicalrep_write_update should be unchanged by this patch
======
src/test/regress/sql/subscription.sql
nitpick = the comment "include_generated_columns and copy_data = true
are mutually exclusive" is not necessary because this all falls under
the existing comment "fail - invalid option combinations"
nitpick - let's explicitly put "copy_data = true" in the CREATE
SUBSCRIPTION to make it more obvious
======
99. Please also refer to the attached 'diffs' patch which implements
all of my nitpicks issues mentioned above.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index a296305..f7c57d4 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -517,7 +517,7 @@ CREATE TABLE people (
Generated columns may be skipped during logical replication according to the
<command>CREATE SUBSCRIPTION</command> option
<link linkend="sql-createsubscription-params-with-include-generated-columns">
- <literal>include_generated_columns</literal></link>,
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5ff5078..52f1551 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -164,7 +164,7 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns should be publicated, or NULL if all columns are included
+ * Columns to be published, or NULL if all columns are included
* implicitly. This bitmap only considers the column list of the
* publication and include_generated_columns option: other reasons should
* be checked at user side. Note that the attnums in this bitmap are not
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0bb5782..50c5911 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -160,7 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
- bool includegencols; /* Publish generated columns data */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index b9a64d9..c409638 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -230,8 +230,7 @@ extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary,
- Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 36916c0..1a99099 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,8 +99,7 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - include_generated_columns and copy_data = true are mutually exclusive
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7944152..7922dfd 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,9 +59,7 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-
--- fail - include_generated_columns and copy_data = true are mutually exclusive
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
Attachments:
[text/plain] PS_NITPICKS_20240705_GENCOLS_V150001.txt (5.1K, ../../CAHut+Pv50FoqfM7f-3SJ441UDzLfG9LqgggLm1Cm0DcGG6VCnQ@mail.gmail.com/2-PS_NITPICKS_20240705_GENCOLS_V150001.txt)
download | inline diff:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index a296305..f7c57d4 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -517,7 +517,7 @@ CREATE TABLE people (
Generated columns may be skipped during logical replication according to the
<command>CREATE SUBSCRIPTION</command> option
<link linkend="sql-createsubscription-params-with-include-generated-columns">
- <literal>include_generated_columns</literal></link>,
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5ff5078..52f1551 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -164,7 +164,7 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns should be publicated, or NULL if all columns are included
+ * Columns to be published, or NULL if all columns are included
* implicitly. This bitmap only considers the column list of the
* publication and include_generated_columns option: other reasons should
* be checked at user side. Note that the attnums in this bitmap are not
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0bb5782..50c5911 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -160,7 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
- bool includegencols; /* Publish generated columns data */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index b9a64d9..c409638 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -230,8 +230,7 @@ extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary,
- Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 36916c0..1a99099 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,8 +99,7 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
--- fail - include_generated_columns and copy_data = true are mutually exclusive
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7944152..7922dfd 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,9 +59,7 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-
--- fail - include_generated_columns and copy_data = true are mutually exclusive
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-08 07:50 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-08 07:50 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shlok, Here are some review comments for patch v15-0003.
======
src/backend/catalog/pg_publication.c
1. publication_translate_columns
The function comment says:
* Translate a list of column names to an array of attribute numbers
* and a Bitmapset with them; verify that each attribute is appropriate
* to have in a publication column list (no system or generated attributes,
* no duplicates). Additional checks with replica identity are done later;
* see pub_collist_contains_invalid_column.
That part about "[no] generated attributes" seems to have gone stale
-- e.g. not quite correct anymore. Should it say no VIRTUAL generated
attributes?
======
src/backend/replication/logical/proto.c
2. logicalrep_write_tuple and logicalrep_write_attrs
I thought all the code fragments like this:
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
don't need to be in the code anymore, because of the BitMapSet (BMS)
processing done to make the "column list" for publication where
disallowed generated cols should already be excluded from the BMS,
right?
So shouldn't all these be detected by the following statement:
if (!column_in_column_list(att->attnum, columns))
continue;
======
src/backend/replication/logical/tablesync.c
3.
+ if(server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 170000 &&
MySubscription->includegencols;
+
+ if (gencols_allowed)
+ {
Should say server_version >= 180000, instead of 170000
======
src/backend/replication/pgoutput/pgoutput.c
4. send_relation_and_attrs
(this is a similar comment for #2 above)
IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
process the generated columns up-front means there is no need to check
them again in code like this.
They should be discovered anyway in the subsequent check:
/* Skip this attribute if it's not present in the column list */
if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
======
src/test/subscription/t/011_generated.pl
5.
AFAICT there are still multiple comments (e.g. for the "TEST tab<n>"
comments) where it still says "generated" instead of "stored
generated". I did not make a "nitpicks" diff for these because those
comments are inherited from the prior patch 0002 which still has
outstanding review comments on it too. Please just search/replace
them.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-08 12:04 Shlok Kyal <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Shlok Kyal @ 2024-07-08 12:04 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, 8 Jul 2024 at 13:20, Peter Smith <[email protected]> wrote:
>
> Hi Shlok, Here are some review comments for patch v15-0003.
>
> ======
> src/backend/catalog/pg_publication.c
>
> 1. publication_translate_columns
>
> The function comment says:
> * Translate a list of column names to an array of attribute numbers
> * and a Bitmapset with them; verify that each attribute is appropriate
> * to have in a publication column list (no system or generated attributes,
> * no duplicates). Additional checks with replica identity are done later;
> * see pub_collist_contains_invalid_column.
>
> That part about "[no] generated attributes" seems to have gone stale
> -- e.g. not quite correct anymore. Should it say no VIRTUAL generated
> attributes?
Yes, we should use VIRTUAL generated attributes, I have modified it.
> ======
> src/backend/replication/logical/proto.c
>
> 2. logicalrep_write_tuple and logicalrep_write_attrs
>
> I thought all the code fragments like this:
>
> + if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
> + continue;
> +
>
> don't need to be in the code anymore, because of the BitMapSet (BMS)
> processing done to make the "column list" for publication where
> disallowed generated cols should already be excluded from the BMS,
> right?
>
> So shouldn't all these be detected by the following statement:
> if (!column_in_column_list(att->attnum, columns))
> continue;
The current BMS logic do not handle the Virtual Generated Columns.
There can be cases where we do not want a virtual generated column but
it would be present in BMS.
To address this I have added the above logic. I have added this logic
similar to the checks of 'attr->attisdropped'.
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 4. send_relation_and_attrs
>
> (this is a similar comment for #2 above)
>
> IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
> process the generated columns up-front means there is no need to check
> them again in code like this.
>
> They should be discovered anyway in the subsequent check:
> /* Skip this attribute if it's not present in the column list */
> if (columns != NULL && !bms_is_member(att->attnum, columns))
> continue;
Same explanation as above.
I have addressed all the comments in v16-0003 patch. Please refer [1].
[1]: https://www.postgresql.org/message-id/CANhcyEXw%3DBFFVUqohWES9EPkdq-ZMC5QRBVQqQPzrO%3DQ7uzFQw%40mail...
Thanks and Regards,
Shlok Kyal
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-09 04:22 Peter Smith <[email protected]>
parent: Shlok Kyal <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-09 04:22 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shlok, here are my review comments for v16-0003.
======
src/backend/replication/logical/proto.c
On Mon, Jul 8, 2024 at 10:04 PM Shlok Kyal <[email protected]> wrote:
>
> On Mon, 8 Jul 2024 at 13:20, Peter Smith <[email protected]> wrote:
> >
> >
> > 2. logicalrep_write_tuple and logicalrep_write_attrs
> >
> > I thought all the code fragments like this:
> >
> > + if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
> > + continue;
> > +
> >
> > don't need to be in the code anymore, because of the BitMapSet (BMS)
> > processing done to make the "column list" for publication where
> > disallowed generated cols should already be excluded from the BMS,
> > right?
> >
> > So shouldn't all these be detected by the following statement:
> > if (!column_in_column_list(att->attnum, columns))
> > continue;
> The current BMS logic do not handle the Virtual Generated Columns.
> There can be cases where we do not want a virtual generated column but
> it would be present in BMS.
> To address this I have added the above logic. I have added this logic
> similar to the checks of 'attr->attisdropped'.
>
Hmm. I thought the BMS idea of patch 0001 is to discover what columns
should be replicated up-front. If they should not be replicated (e.g.
virtual generated columns cannot be) then they should never be in the
BMS.
So what you said ("There can be cases where we do not want a virtual
generated column but it would be present in BMS") should not be
happening. If that is happening then it sounds more like a bug in the
new BMS logic of pgoutput_column_list_init() function. In other words,
if what you say is true, then it seems like the current extra
conditions you have in patch 0004 are just a band-aid to cover a
problem of the BMS logic of patch 0001. Am I mistaken?
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 4. send_relation_and_attrs
> >
> > (this is a similar comment for #2 above)
> >
> > IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
> > process the generated columns up-front means there is no need to check
> > them again in code like this.
> >
> > They should be discovered anyway in the subsequent check:
> > /* Skip this attribute if it's not present in the column list */
> > if (columns != NULL && !bms_is_member(att->attnum, columns))
> > continue;
> Same explanation as above.
As above.
======
src/test/subscription/t/011_generated.pl
I'm not sure if you needed to say "STORED" generated cols for the
subscriber-side columns but anyway, whatever is done needs to be done
consistently. FYI, below you did *not* say STORED for subscriber-side
generated cols, but in other comments for subscriber-side generated
columns, you did say STORED.
# tab3:
# publisher-side tab3 has STORED generated col 'b' but
# subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
~
# tab4:
# publisher-side tab4 has STORED generated cols 'b' and 'c' but
# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
# where columns on publisher/subscriber are in a different order
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-11 06:12 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 3 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-07-11 06:12 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Jul 8, 2024 at 10:53 AM Peter Smith <[email protected]> wrote:
>
> Here are review comments for v15-0001
>
> ======
> doc/src/sgml/ddl.sgml
>
> nitpick - there was a comma (,) which should be a period (.)
>
> ======
> .../libpqwalreceiver/libpqwalreceiver.c
>
> 1.
> + if (options->proto.logical.include_generated_columns &&
> + PQserverVersion(conn->streamConn) >= 170000)
> + appendStringInfoString(&cmd, ", include_generated_columns 'true'");
> +
>
> Should now say >= 180000
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> nitpick - comment wording for RelationSyncEntry.collist.
>
> ~~
>
> 2.
> pgoutput_column_list_init:
>
> I found the current logic to be quite confusing. I assume the code is
> working OK, because AFAIK there are plenty of tests and they are all
> passing, but the logic seems somewhat repetitive and there are also no
> comments to explain it adding to my confusion.
>
> IIUC, PRIOR TO THIS PATCH:
>
> BMS field 'columns' represented the "columns of the column list" or it
> was NULL if there was no publication column list (and it was also NULL
> if the column list contained every column).
>
> IIUC NOW, WITH THIS PATCH:
>
> The BMS field 'columns' meaning is changed slightly to be something
> like "columns to be replicated" or NULL if all columns are to be
> replicated. This is almost the same thing except we are now handing
> the generated columns up-front, so generated columns will or won't
> appear in the BMS according to the "include_generated_columns"
> parameter. See how this is all a bit subtle which is why copious new
> comments are required to explain it...
>
> So, although the test result evidence suggests this is working OK, I
> have many questions/issues about it. Here are some to start with:
>
> 2a. It needs a lot more (summary and detailed) comments explaining the
> logic now that the meaning is slightly different.
>
> 2b. What is the story with the FOR ALL TABLES case now? Previously,
> there would always be NULL 'columns' for "FOR ALL TABLES" case -- the
> comment still says so. But now you've tacked on a 2nd pass of
> iterations to build the BMS outside of the "if (!pub->alltables)"
> check. Is that OK?
>
> 2c. The following logic seemed unexpected:
> - if (bms_num_members(cols) == nliveatts)
> + if (bms_num_members(cols) == nliveatts &&
> + data->include_generated_columns)
> {
> bms_free(cols);
> cols = NULL;
> `
> I had thought the above code would look different -- more like:
> if (att->attgenerated && !data->include_generated_columns)
> continue;
>
> nliveatts++;
> ...
>
> 2d. Was so much duplicated code necessary? It feels like the whole
> "Get the number of live attributes." and assignment of cols to NULL
> might be made common to both code paths.
>
> 2e. I'm beginning to question the pros/cons of the new BMS logic; I
> had suggested trying this way (processing the generated columns
> up-front in the BMS 'columns' list) to reduce patch code and simplify
> all the subsequent API delegation of "include_generated_cloumns"
> everywhere like it was in v14-0001. Indeed, that part was a success
> and the patch is now smaller. But I don't like much that we've traded
> reduced code overall for increased confusing code in that BMS
> function. If all this BMS code can be refactored and commented to be
> easier to understand then maybe all will be well, but if it can't then
> maybe this BMS change was a bridge too far. I haven't given up on it
> just yet, but I wonder what was your opinion about it, and do other
> people have thoughts about whether this was the good direction to
> take?
I have created a separate patch(v17-0004) for this idea. Will address
this comment in the next version of patches.
> ======
> src/bin/pg_dump/pg_dump.c
>
> 3.
> + if (fout->remoteVersion >= 170000)
> + appendPQExpBufferStr(query,
> + " s.subincludegencols\n");
> + else
> + appendPQExpBufferStr(query,
> + " false AS subincludegencols\n");
>
> Should now say >= 180000
>
> ======
> src/bin/psql/describe.c
>
> 4.
> + /* include_generated_columns is only supported in v18 and higher */
> + if (pset.sversion >= 170000)
> + appendPQExpBuffer(&buf,
> + ", subincludegencols AS \"%s\"\n",
> + gettext_noop("Include generated columns"));
> +
>
> Should now say >= 180000
>
> ======
> src/include/catalog/pg_subscription.h
>
> nitpick - let's make the comment the same as in WalRcvStreamOptions
>
> ======
> src/include/replication/logicalproto.h
>
> nitpick - extern for logicalrep_write_update should be unchanged by this patch
>
> ======
> src/test/regress/sql/subscription.sql
>
> nitpick = the comment "include_generated_columns and copy_data = true
> are mutually exclusive" is not necessary because this all falls under
> the existing comment "fail - invalid option combinations"
>
> nitpick - let's explicitly put "copy_data = true" in the CREATE
> SUBSCRIPTION to make it more obvious
>
> ======
> 99. Please also refer to the attached 'diffs' patch which implements
> all of my nitpicks issues mentioned above.
The attached Patches contain all the suggested changes. Here, v17-0001
is modified to fix the comments, v17-0002 and v17-0003 are modified
according to the changes in v17-0001 patch and v17-0004 patch contains
the changes related to Bitmapset(BMS) idea that changes the 'columns'
meaning to include generated cols also where necessary.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v17-0002-Support-replication-of-generated-column-during-i.patch (26.3K, ../../CAHv8RjJ0gAUd62PvBRXCPYy2oTNZWEY-Qe8cBNzQaJPVMZCeGA@mail.gmail.com/2-v17-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From f4b52e6a82d9d1877d774ed16db6d836858226b6 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Wed, 10 Jul 2024 15:41:54 +0530
Subject: [PATCH v17 2/4] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is
replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when
'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Summary:
when (include_generated_columns = true)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column: The
publisher generated column value is copied.
* publisher generated column => subscriber generated column: The
publisher generated column value is not copied. The subscriber
generated column will be filled with the subscriber-side computed or
default data.
when (include_generated_columns = false)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column:
Publisher generated column is not replicated. The subscriber column
will be filled with the subscriber-side default data.
* publisher generated column => subscriber generated column: Publisher
generated column is not replicated. The subscriber generated column
will be filed with the subscriber-side computed or default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 --
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/tablesync.c | 135 ++++++++++++++++----
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 2 -
src/test/regress/sql/subscription.sql | 1 -
src/test/subscription/t/004_sync.pl | 56 ++++++++
src/test/subscription/t/011_generated.pl | 135 +++++++++++++++++++-
9 files changed, 300 insertions(+), 52 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index f942b58565..408a9157ec 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..9de0b75330 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index b00267f042..1edba12a36 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -692,21 +693,68 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create list of columns for COPY based on logical relation mapping. Do not
+ * include generated columns of the subscription table in the column list.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *localgenlist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
+
+ /*
+ * This loop checks for generated columns on subscription table.
+ */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int remote_attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (remote_attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if (!remotegenlist[remote_attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'localgenlist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ localgenlist[remote_attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY, excluding columns that are
+ * subscription table generated columns.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if (!localgenlist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
+ pfree(localgenlist);
return attnamelist;
}
@@ -791,19 +839,21 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
+ int server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
lrel->nspname = nspname;
lrel->relname = relname;
@@ -851,7 +901,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
WalRcvExecResult *pubres;
TupleTableSlot *tslot;
@@ -948,18 +998,27 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey)");
+
+ if(server_version >= 120000)
+ appendStringInfo(&cmd, ", a.attgenerated != ''");
+
+ appendStringInfo(&cmd,
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if ((server_version >= 120000 && server_version < 180000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1032,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1065,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1077,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist_res = remotegenlist;
walrcv_clear_result(res);
/*
@@ -1037,7 +1099,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* 3) one of the subscribed publications is declared as TABLES IN SCHEMA
* that includes this relation
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
StringInfoData pub_names;
@@ -1123,10 +1185,13 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
+ bool gencol_copy_needed = false;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1200,29 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /* Check if remote column list has any generated column */
+ if(MySubscription->includegencols)
+ {
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if(remotegenlist[i])
+ {
+ gencol_copy_needed = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * Regular table with no row filter and copy of generated columns is
+ * not necessary.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL && !gencol_copy_needed)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1173,13 +1256,20 @@ copy_table(Relation rel)
* (SELECT ...), but we can't just do SELECT * because we need to not
* copy generated columns. For tables with any row filters, build a
* SELECT query with OR'ed row filters for COPY.
+ *
+ * We also need to use this same COPY (SELECT ...) syntax when
+ * 'include_generated_columns' is specified as true and the remote
+ * table has generated columns, because copy of generated columns is
+ * not supported by the normal COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_node(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1327,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 1a990993ba..592c1f0667 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,8 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7922dfd3cd..8c7381fbfc 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,7 +59,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index a2d9462395..62462c0c95 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -172,6 +172,62 @@ ok( $node_publisher->poll_query_until(
'postgres', 'SELECT count(*) = 0 FROM pg_replication_slots'),
'DROP SUBSCRIPTION during error can clean up the slots on the publisher');
+# clean up
+$node_publisher->safe_psql('postgres', "DROP TABLE tab_rep");
+$node_subscriber->safe_psql('postgres', "DROP TABLE tab_rep");
+
+#
+# TEST CASE:
+#
+# When a subscription table has a column missing that was specified on
+# the publication table.
+#
+
+# setup structure with existing data on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_rep (a int, b int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_rep VALUES (1, 1), (2, 2), (3, 3)");
+
+# add table on subscriber; note column 'b' is missing
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_rep (a int)");
+
+my $offset = -s $node_subscriber->logfile;
+
+# create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+);
+
+# check for missing column error
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_rep" is missing replicated column: "b"/,
+ $offset);
+
+# clean up
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub");
+$node_subscriber->safe_psql('postgres', "DROP TABLE tab_rep");
+
+#
+# TEST CASE:
+#
+# When a subscription table has a generated column corresponding to a
+# non-generated column on publication table
+#
+
+# create table on subscriber side with generated column 'b'
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_rep (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+
+# create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+);
+
+# check for generated column mismatch error
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_rep" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f4499691cb..9e26373c43 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -16,6 +16,8 @@ $node_publisher->start;
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 10");
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -41,6 +43,33 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+# tab4:
+# publisher-side tab4 has generated cols 'b' and 'c' but
+# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
+# where columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab4 (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab5:
+# publisher-side tab5 has non-generated col 'b' but
+# subscriber-side tab5 has generated col 'b'
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# tab6:
+# tables for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -49,6 +78,12 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab5 (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR TABLE tab1");
@@ -56,15 +91,22 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub2 FOR TABLE tab2");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub4 FOR TABLE tab4");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub5 FOR TABLE tab5");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr' PUBLICATION pub4 WITH (include_generated_columns = true)"
);
# Wait for initial sync of all subscriptions
@@ -76,10 +118,24 @@ is( $result, qq(1|22
3|66), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'generated column initial sync');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(), 'unsubscribed table initial data');
# data to replicate
@@ -104,7 +160,10 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
@@ -119,11 +178,75 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub3');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
-is( $result, qq(4|24
+is( $result, qq(1|21
+2|22
+3|23
+4|24
5|25),
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
+#
+# TEST tab4: the publisher-side cols 'b' and 'c' are generated and subscriber-side
+# col 'b' is not generated and col 'c' is generated. So confirmed that the different
+# order of columns on subscriber-side replicate data to correct columns.
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub4');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated columns with different order on subscriber');
+
+#
+# TEST tab5: publisher-side col 'b' is not-generated and subscriber-side col 'b'
+# is generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
+# The subscription sub5 is created here, instead of earlier with the other subscriptions,
+# because sub5 will cause the tablesync worker to restart repetitively.
+#
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub5 CONNECTION '$publisher_connstr' PUBLICATION pub5 WITH (include_generated_columns = true)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab5" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub5");
+
+#
+# TEST tab6: After ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub4 ADD TABLE tab6");
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub4 REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('sub4');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'add new table to existing publication');
+
+#
+# TEST tab6: Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (4), (5)");
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
--
2.41.0.windows.3
[application/octet-stream] v17-0001-Enable-support-for-include_generated_columns-opt.patch (89.6K, ../../CAHv8RjJ0gAUd62PvBRXCPYy2oTNZWEY-Qe8cBNzQaJPVMZCeGA@mail.gmail.com/3-v17-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 220f72f46184223ab1da46ea4dbce75032e7ba76 Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Mon, 20 May 2024 10:58:31 +0530
Subject: [PATCH v17 1/4] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
-- Using Create Subscription
CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=postgres' PUBLICATION pub2 WITH
(include_generated_columns = true, copy_data = false);
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 43 +++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 64 ++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 443 insertions(+), 138 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..9ecd4fa0b7 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..eab4aa68c9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index c5e11a6699..f7c57d47af 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..226c3641b9 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6524,8 +6535,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..f942b58565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6c42c209d2..a762051732 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 95c09c9516..7405eb3deb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3b285894db..3e6d68a3d6 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..6bc9f9d403 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -166,6 +167,8 @@ typedef struct RelationSyncEntry
/*
* Columns included in the publication, or NULL if all columns are
* included implicitly. Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
@@ -283,11 +286,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +401,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +746,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +766,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +783,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +802,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1105,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1551,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..e99f528e39 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5167,6 +5177,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..f7b8d59413 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..50c5911d23 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..9275b3a617 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..1a990993ba 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +376,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +388,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +401,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +417,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..7922dfd3cd 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..f4499691cb 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,16 +28,44 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
+
+# tab3:
+# publisher-side tab3 has generated col 'b' but
+# subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+);
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +75,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is($result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is($result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,6 +96,34 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#
+# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is not generated, so confirm that col 'b' IS replicated.
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub2');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+);
+
+#
+# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
+# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
+# can know this because the result value is the subscriber-side computation
+# (which is not the same as the publisher-side computation for col 'b').
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub3');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
[application/octet-stream] v17-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (14.4K, ../../CAHv8RjJ0gAUd62PvBRXCPYy2oTNZWEY-Qe8cBNzQaJPVMZCeGA@mail.gmail.com/4-v17-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From 1ae46b5b14c4f91facba6036f02c29b2dbd9d902 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Wed, 10 Jul 2024 16:03:33 +0530
Subject: [PATCH v17 3/4] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
.../expected/generated_columns.out | 1 +
.../test_decoding/sql/generated_columns.sql | 4 +-
contrib/test_decoding/test_decoding.c | 15 ++++++-
doc/src/sgml/ref/create_subscription.sgml | 4 +-
src/backend/catalog/pg_publication.c | 12 ++++++
src/backend/replication/logical/proto.c | 40 +++++++++++++++----
src/backend/replication/logical/tablesync.c | 21 ++++++++--
src/backend/replication/pgoutput/pgoutput.c | 5 ++-
src/test/subscription/t/011_generated.pl | 28 ++++++-------
9 files changed, 98 insertions(+), 32 deletions(-)
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index f3b26aa9e1..a79510705c 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -50,3 +50,4 @@ SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
stop
(1 row)
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 6d6d1d6564..997cdebc7e 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -19,4 +19,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..a847050f6e 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,8 +557,19 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
- continue;
+ if (attr->attgenerated)
+ {
+ /*
+ * Don't print generated columns when
+ * 'include_generated_columns' is false.
+ */
+ if (!include_generated_columns)
+ continue;
+
+ /* Don't print generated columns unless they are STORED. */
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
/*
* Don't print system columns, oid will already have been printed if
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..1809e140ea 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -520,6 +521,7 @@ publication_translate_columns(Relation targetrel, List *columns,
{
char *colname = strVal(lfirst(lc));
AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+ Form_pg_attribute att;
if (attnum == InvalidAttrNumber)
ereport(ERROR,
@@ -533,6 +535,13 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ att = TupleDescAttr(tupdesc, attnum - 1);
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("generated column \"%s\" is not STORED so cannot be used in a publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1228,6 +1237,9 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
attnums[nattnums++] = att->attnum;
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 7405eb3deb..1c35fb6cff 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,8 +793,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -817,8 +823,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -957,8 +969,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -981,8 +999,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1edba12a36..c2a7d18774 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -714,7 +714,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
int remote_attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (!attr->attgenerated)
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
@@ -1008,11 +1008,24 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped", lrel->remoteid);
+ " AND NOT a.attisdropped", lrel->remoteid);
- if ((server_version >= 120000 && server_version < 180000) ||
- !MySubscription->includegencols)
+ if(server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
+
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ /* TODO: use ATTRIBUTE_GENERATED_VIRTUAL*/
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
+ {
+ /* Replication of generated cols is not supported. */
appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+ }
appendStringInfo(&cmd,
" AND a.attrelid = %u"
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6bc9f9d403..944554d5d7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -786,7 +786,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -1108,6 +1108,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 9e26373c43..1c49f4172c 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -30,22 +30,22 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+# publisher-side tab2 has stored generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
# tab3:
-# publisher-side tab3 has generated col 'b' but
-# subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has stored generated col 'b' but
+# subscriber-side tab3 has DIFFERENT COMPUTATION stored generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
# tab4:
-# publisher-side tab4 has generated cols 'b' and 'c' but
-# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
+# publisher-side tab4 has stored generated cols 'b' and 'c' but
+# subscriber-side tab4 has non-generated col 'b', and stored generated-col 'c'
# where columns on publisher/subscriber are in a different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -56,7 +56,7 @@ $node_subscriber->safe_psql('postgres',
# tab5:
# publisher-side tab5 has non-generated col 'b' but
-# subscriber-side tab5 has generated col 'b'
+# subscriber-side tab5 has stored generated col 'b'
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
@@ -153,7 +153,7 @@ is( $result, qq(1|22|
6|132|), 'generated columns replicated');
#
-# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
+# TEST tab2: the publisher-side col 'b' is stored generated, and the subscriber-side
# col 'b' is not generated, so confirm that col 'b' IS replicated.
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
@@ -169,8 +169,8 @@ is( $result, qq(1|2
);
#
-# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
-# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
+# TEST tab3: the publisher-side col 'b' is stored generated, and the subscriber-side
+# col 'b' is also stored generated, so confirmed that col 'b' IS NOT replicated. We
# can know this because the result value is the subscriber-side computation
# (which is not the same as the publisher-side computation for col 'b').
#
@@ -187,8 +187,8 @@ is( $result, qq(1|21
);
#
-# TEST tab4: the publisher-side cols 'b' and 'c' are generated and subscriber-side
-# col 'b' is not generated and col 'c' is generated. So confirmed that the different
+# TEST tab4: the publisher-side cols 'b' and 'c' are stored generated and subscriber-side
+# col 'b' is not generated and col 'c' is stored generated. So confirmed that the different
# order of columns on subscriber-side replicate data to correct columns.
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
@@ -204,7 +204,7 @@ is( $result, qq(1|2|22
#
# TEST tab5: publisher-side col 'b' is not-generated and subscriber-side col 'b'
-# is generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
+# is stored generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
# The subscription sub5 is created here, instead of earlier with the other subscriptions,
# because sub5 will cause the tablesync worker to restart repetitively.
#
@@ -232,8 +232,8 @@ is( $result, qq(1|2|22
3|6|66), 'add new table to existing publication');
#
-# TEST tab6: Drop the generated column's expression on subscriber side.
-# This changes the generated column into a non-generated column.
+# TEST tab6: Drop the stored generated column's expression on subscriber side.
+# This changes the stored generated column into a non-generated column.
#
$node_subscriber->safe_psql('postgres',
"ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
--
2.41.0.windows.3
[application/octet-stream] v17-0004-Improve-include-generated-column-option-handling.patch (14.1K, ../../CAHv8RjJ0gAUd62PvBRXCPYy2oTNZWEY-Qe8cBNzQaJPVMZCeGA@mail.gmail.com/5-v17-0004-Improve-include-generated-column-option-handling.patch)
download | inline diff:
From de5930bf4440dc8c03c9b96a29ef13596872c389 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Thu, 11 Jul 2024 10:11:00 +0530
Subject: [PATCH v17 4/4] Improve include generated column option handling by
using bms
Improve include generated column option handling by using bms
---
src/backend/replication/logical/proto.c | 44 +++------
src/backend/replication/pgoutput/pgoutput.c | 102 ++++++++++++++++----
src/include/replication/logicalproto.h | 12 +--
3 files changed, 99 insertions(+), 59 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 1c35fb6cff..26956a54ab 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,12 +30,10 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -414,8 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -427,8 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -461,8 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -483,13 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -539,7 +532,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -559,8 +552,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
/*
@@ -676,7 +668,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
char *relname;
@@ -698,7 +690,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns, include_generated_columns);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -775,8 +767,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -795,8 +786,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
@@ -825,8 +814,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
@@ -950,8 +937,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
- bool include_generated_columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -971,8 +957,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
@@ -1001,8 +985,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 944554d5d7..19b6d4e7e8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,8 +86,7 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -165,8 +164,10 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns included in the publication, or NULL if all columns are
- * included implicitly. Note that the attnums in this bitmap are not
+ * Columns should be publicated, or NULL if all columns are included
+ * implicitly. This bitmap only considers the column list of the
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* publication and include_generated_columns option: other reasons should
* be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
@@ -746,13 +747,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -766,7 +765,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -786,9 +785,6 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
- continue;
-
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -802,7 +798,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -1028,6 +1024,34 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Prepare new column list bitmap. This includes all the columns of the table.
+ */
+static Bitmapset *
+prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
+ TupleDesc desc)
+{
+ Bitmapset *cols = NULL;
+ MemoryContext oldcxt = NULL;
+
+ pgoutput_ensure_entry_cxt(data, entry);
+ oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped)
+ continue;
+
+ cols = bms_add_member(cols, i + 1);
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+
+ return cols;
+}
+
/*
* Initialize the column list.
*/
@@ -1118,7 +1142,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
* If column list includes all the columns of the table,
* set it to NULL.
*/
- if (bms_num_members(cols) == nliveatts)
+ if (bms_num_members(cols) == nliveatts &&
+ data->include_generated_columns)
{
bms_free(cols);
cols = NULL;
@@ -1129,6 +1154,46 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
}
}
+ /* Do additional checks if the generated columns must be replicated */
+ if (!data->include_generated_columns)
+ {
+ TupleDesc desc = RelationGetDescr(relation);
+ int nliveatts = 0;
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped */
+ if (att->attisdropped)
+ continue;
+
+ /* Count all valid attributes */
+ nliveatts++;
+
+ /* Skip if the attribute is not generated */
+ if (!att->attgenerated)
+ continue;
+
+ /* Prepare new bms if not allocated yet */
+ if (cols == NULL)
+ cols = prepare_all_columns_bms(data, entry, desc);
+
+ /* Delete the corresponding column from the bms */
+ cols = bms_del_member(cols, i + 1);
+ }
+
+ /*
+ * If column list includes all the columns of the table, set it to
+ * NULL.
+ */
+ if (bms_num_members(cols) == nliveatts)
+ {
+ bms_free(cols);
+ cols = NULL;
+ }
+ }
+
if (first)
{
entry->columns = cols;
@@ -1554,18 +1619,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns,
- data->include_generated_columns);
+ new_slot, data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
default:
Assert(false);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 34ec40b07e..b9a64d9c95 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,22 +225,19 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
TupleTableSlot *newslot, bool binary,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -251,8 +248,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns,
- bool include_generated_columns);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
--
2.41.0.windows.3
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-12 06:43 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
2 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-12 06:43 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham.
Thanks for separating the new BMS 'columns' modification.
Here are my review comments for the latest patch v17-0001.
======
1. src/backend/replication/pgoutput/pgoutput.c
/*
* Columns included in the publication, or NULL if all columns are
* included implicitly. Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
With this latest 0001 there is now no change to the original
interpretation of RelationSyncEntry BMS 'columns'. So, I think this
field comment should remain unchanged; i.e. it should be the same as
the current HEAD comment.
======
src/test/subscription/t/011_generated.pl
nitpick - comment changes for 'tab2' and 'tab3' to make them more consistent.
======
99.
Please refer to the attached diff patch which implements any nitpicks
described above.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f449969..fe32987 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,14 +28,16 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+# tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
# tab3:
-# publisher-side tab3 has generated col 'b' but
-# subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
@@ -97,8 +99,11 @@ is( $result, qq(1|22|
6|132|), 'generated columns replicated');
#
-# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
-# col 'b' is not generated, so confirm that col 'b' IS replicated.
+# TEST tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
+#
+# Confirm that col 'b' is replicated.
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
@@ -110,10 +115,13 @@ is( $result, qq(4|8
);
#
-# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
-# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
-# can know this because the result value is the subscriber-side computation
-# (which is not the same as the publisher-side computation for col 'b').
+# TEST tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
+#
+# Confirm that col 'b' is NOT replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub3');
Attachments:
[text/plain] PS_NITPICKS_GENCOLS_V170001.txt (2.5K, ../../CAHut+PtVfrbx0jb42LCmS=-LcMTtWxm+vhaoArkjg7Z0mvuXbg@mail.gmail.com/2-PS_NITPICKS_GENCOLS_V170001.txt)
download | inline diff:
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f449969..fe32987 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,14 +28,16 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side tab2 has generated col 'b' but subscriber-side tab2 has NON-generated col 'b'.
+# tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
# tab3:
-# publisher-side tab3 has generated col 'b' but
-# subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
@@ -97,8 +99,11 @@ is( $result, qq(1|22|
6|132|), 'generated columns replicated');
#
-# TEST tab2: the publisher-side col 'b' is generated, and the subscriber-side
-# col 'b' is not generated, so confirm that col 'b' IS replicated.
+# TEST tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
+#
+# Confirm that col 'b' is replicated.
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
@@ -110,10 +115,13 @@ is( $result, qq(4|8
);
#
-# TEST tab3: the publisher-side col 'b' is generated, and the subscriber-side
-# col 'b' is also generated, so confirmed that col 'b' IS NOT replicated. We
-# can know this because the result value is the subscriber-side computation
-# (which is not the same as the publisher-side computation for col 'b').
+# TEST tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
+#
+# Confirm that col 'b' is NOT replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub3');
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-15 02:38 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
2 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-15 02:38 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, here are some review comments about patch v17-0003
======
1.
Missing a docs change?
Previously, (v16-0002) the patch included a change to
doc/src/sgml/protocol.sgml like below to say STORED generated instead
of just generated.
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal>
generated columns
+ should be included in the string representation of tuples
during logical
+ decoding in PostgreSQL.
</para>
Why is that v16 change no longer present in patch v17-0003?
======
src/backend/catalog/pg_publication.c
2.
Previously, (v16-0003) this patch included a change to clarify the
kind of generated cols that are allowed in a column list.
* Translate a list of column names to an array of attribute numbers
* and a Bitmapset with them; verify that each attribute is appropriate
- * to have in a publication column list (no system or generated attributes,
- * no duplicates). Additional checks with replica identity are done later;
- * see pub_collist_contains_invalid_column.
+ * to have in a publication column list (no system or virtual generated
+ * attributes, no duplicates). Additional checks with replica identity
+ * are done later; see pub_collist_contains_invalid_column.
Why is that v16 change no longer present in patch v17-0003?
======
src/backend/replication/logical/tablesync.c
3. make_copy_attnamelist
- if (!attr->attgenerated)
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
IIUC this logic is checking to make sure the subscriber-side table
column was not a generated column (because we don't replicate on top
of generated columns). So, does the distinction of STORED/VIRTUAL
really matter here?
~~~
fetch_remote_table_info:
nitpick - Should not change any spaces unrelated to the patch
======
send_relation_and_attrs:
- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated !=
ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
continue;
nitpick - It seems over-complicated. Conditions can be split so the
code fragment looks the same as in other places in this patch.
======
99.
Please see the attached diffs patch that implements any nitpicks
mentioned above.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index c2a7d18..5288769 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1008,7 +1008,7 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped", lrel->remoteid);
+ " AND NOT a.attisdropped", lrel->remoteid);
if(server_version >= 120000)
{
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 944554d..a256ab7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -786,8 +786,14 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (att->atttypid < FirstGenbkiObjectId)
continue;
Attachments:
[text/plain] PS_NITPICKS_20240715_GENCOLS_V170003.txt (1.3K, ../../CAHut+PtqH6kA5ifH8zcLLatknY=hnZWr6vXJ8x7rzFm3HFLkMA@mail.gmail.com/2-PS_NITPICKS_20240715_GENCOLS_V170003.txt)
download | inline diff:
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index c2a7d18..5288769 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1008,7 +1008,7 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped", lrel->remoteid);
+ " AND NOT a.attisdropped", lrel->remoteid);
if(server_version >= 120000)
{
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 944554d..a256ab7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -786,8 +786,14 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && (att->attgenerated != ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (att->atttypid < FirstGenbkiObjectId)
continue;
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-15 05:39 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
2 siblings, 0 replies; 60+ messages in thread
From: Peter Smith @ 2024-07-15 05:39 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, I had a quick look at the patch v17-0004 which is the split-off
new BMS logic.
IIUC this 0004 is currently undergoing some refactoring and
cleaning-up, so I won't comment much about it except to give the
following observation below.
======
src/backend/replication/logical/proto.c.
I did not expect to see any code fragments that are still checking
generated columns like below:
logicalrep_write_tuple:
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
~
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
~~~
logicalrep_write_attrs:
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
~
if (att->attgenerated)
{
- if (!include_generated_columns)
- continue;
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
continue;
~~~
AFAIK, now checking support of generated columns will be done when the
BMS 'columns' is assigned, so the continuation code will be handled
like this:
if (!column_in_column_list(att->attnum, columns))
continue;
======
BTW there is a subtle but significant difference in this 0004 patch.
IOW, we are introducing a difference between the list of published
columns VERSUS a publication column list. So please make sure that all
code comments are adjusted appropriately so they are not misleading by
calling these "column lists" still.
BEFORE: BMS 'columns' means "columns of the column list" or NULL if
there was no publication column list
AFTER: BMS 'columns' means "columns to be replicated" or NULL if all
columns are to be replicated
======
Kind Regards,
Peter Smith.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-16 08:58 Shlok Kyal <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shlok Kyal @ 2024-07-16 08:58 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Tue, 9 Jul 2024 at 09:53, Peter Smith <[email protected]> wrote:
>
> Hi Shlok, here are my review comments for v16-0003.
>
> ======
> src/backend/replication/logical/proto.c
>
>
> On Mon, Jul 8, 2024 at 10:04 PM Shlok Kyal <[email protected]> wrote:
> >
> > On Mon, 8 Jul 2024 at 13:20, Peter Smith <[email protected]> wrote:
> > >
> > >
> > > 2. logicalrep_write_tuple and logicalrep_write_attrs
> > >
> > > I thought all the code fragments like this:
> > >
> > > + if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
> > > + continue;
> > > +
> > >
> > > don't need to be in the code anymore, because of the BitMapSet (BMS)
> > > processing done to make the "column list" for publication where
> > > disallowed generated cols should already be excluded from the BMS,
> > > right?
> > >
> > > So shouldn't all these be detected by the following statement:
> > > if (!column_in_column_list(att->attnum, columns))
> > > continue;
> > The current BMS logic do not handle the Virtual Generated Columns.
> > There can be cases where we do not want a virtual generated column but
> > it would be present in BMS.
> > To address this I have added the above logic. I have added this logic
> > similar to the checks of 'attr->attisdropped'.
> >
>
> Hmm. I thought the BMS idea of patch 0001 is to discover what columns
> should be replicated up-front. If they should not be replicated (e.g.
> virtual generated columns cannot be) then they should never be in the
> BMS.
>
> So what you said ("There can be cases where we do not want a virtual
> generated column but it would be present in BMS") should not be
> happening. If that is happening then it sounds more like a bug in the
> new BMS logic of pgoutput_column_list_init() function. In other words,
> if what you say is true, then it seems like the current extra
> conditions you have in patch 0004 are just a band-aid to cover a
> problem of the BMS logic of patch 0001. Am I mistaken?
>
We have created a 0004 patch to use the BMS approach. It will be
addressed in the future 0004 patch.
> > > ======
> > > src/backend/replication/pgoutput/pgoutput.c
> > >
> > > 4. send_relation_and_attrs
> > >
> > > (this is a similar comment for #2 above)
> > >
> > > IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
> > > process the generated columns up-front means there is no need to check
> > > them again in code like this.
> > >
> > > They should be discovered anyway in the subsequent check:
> > > /* Skip this attribute if it's not present in the column list */
> > > if (columns != NULL && !bms_is_member(att->attnum, columns))
> > > continue;
> > Same explanation as above.
>
> As above.
>
We have created a 0004 patch to use the BMS approach. It will be
addressed in the future 0004 patch.
> ======
> src/test/subscription/t/011_generated.pl
>
> I'm not sure if you needed to say "STORED" generated cols for the
> subscriber-side columns but anyway, whatever is done needs to be done
> consistently. FYI, below you did *not* say STORED for subscriber-side
> generated cols, but in other comments for subscriber-side generated
> columns, you did say STORED.
>
> # tab3:
> # publisher-side tab3 has STORED generated col 'b' but
> # subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
>
> ~
>
> # tab4:
> # publisher-side tab4 has STORED generated cols 'b' and 'c' but
> # subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
> # where columns on publisher/subscriber are in a different order
>
Fixed
Please find the updated patch v18-0003 patch at [1].
[1]: https://www.postgresql.org/message-id/CANhcyEW3LVJpRPScz6VBa%3DZipEMV7b-u76PDEALNcNDFURCYMA%40mail.g...
Thanks and Regards,
Shok Kyal
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-16 09:04 Shlok Kyal <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shlok Kyal @ 2024-07-16 09:04 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, 15 Jul 2024 at 08:08, Peter Smith <[email protected]> wrote:
>
> Hi, here are some review comments about patch v17-0003
I have addressed the comments in v18-0003 patch [1].
[1]: https://www.postgresql.org/message-id/CANhcyEW3LVJpRPScz6VBa%3DZipEMV7b-u76PDEALNcNDFURCYMA%40mail.g...
Thanks and Regards,
Shlok Kyal
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-16 11:19 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 3 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-07-16 11:19 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Jul 12, 2024 at 12:13 PM Peter Smith <[email protected]> wrote:
>
> Hi Shubham.
>
> Thanks for separating the new BMS 'columns' modification.
>
> Here are my review comments for the latest patch v17-0001.
>
> ======
>
> 1. src/backend/replication/pgoutput/pgoutput.c
>
> /*
> * Columns included in the publication, or NULL if all columns are
> * included implicitly. Note that the attnums in this bitmap are not
> + * publication and include_generated_columns option: other reasons should
> + * be checked at user side. Note that the attnums in this bitmap are not
> * shifted by FirstLowInvalidHeapAttributeNumber.
> */
> Bitmapset *columns;
> With this latest 0001 there is now no change to the original
> interpretation of RelationSyncEntry BMS 'columns'. So, I think this
> field comment should remain unchanged; i.e. it should be the same as
> the current HEAD comment.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> nitpick - comment changes for 'tab2' and 'tab3' to make them more consistent.
>
> ======
> 99.
> Please refer to the attached diff patch which implements any nitpicks
> described above.
The attached Patches contain all the suggested changes.
v19-0001 - Addressed the comments.
v19-0002 - Rebased the Patch.
v19-0003 - Rebased the Patch.
v19-0004- Addressed all the comments related to Bitmapset(BMS).
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v19-0002-Support-replication-of-generated-column-during-i.patch (27.4K, ../../CAHv8Rj+R0cj=z1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A@mail.gmail.com/2-v19-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From 3076b48a329e02db907c1cf1335a7345502e112d Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Tue, 16 Jul 2024 11:11:50 +0530
Subject: [PATCH v19 2/4] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Summary:
when (include_generated_columns = true)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column: The
publisher generated column value is copied.
* publisher generated column => subscriber generated column: The
publisher generated column value is not copied. The subscriber
generated column will be filled with the subscriber-side computed or
default data.
when (include_generated_columns = false)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column:
Publisher generated column is not replicated. The subscriber column
will be filled with the subscriber-side default data.
* publisher generated column => subscriber generated column: Publisher
generated column is not replicated. The subscriber generated column
will be filed with the subscriber-side computed or default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 --
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/tablesync.c | 144 +++++++++++++---
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 2 -
src/test/regress/sql/subscription.sql | 1 -
src/test/subscription/t/004_sync.pl | 36 ++++
src/test/subscription/t/011_generated.pl | 176 +++++++++++++++++++-
9 files changed, 330 insertions(+), 52 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 507c5ef9c1..0847c174c1 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..9de0b75330 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e03e761392..935be7f934 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -692,21 +693,68 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create list of columns for COPY based on logical relation mapping. Do not
+ * include generated columns of the subscription table in the column list.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *localgenlist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
+
+ /*
+ * This loop checks for generated columns on subscription table.
+ */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int remote_attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (remote_attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if (!remotegenlist[remote_attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'localgenlist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ localgenlist[remote_attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY, excluding columns that are
+ * subscription table generated columns.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if (!localgenlist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
+ pfree(localgenlist);
return attnamelist;
}
@@ -791,19 +839,21 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
+ int server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
lrel->nspname = nspname;
lrel->relname = relname;
@@ -851,7 +901,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
WalRcvExecResult *pubres;
TupleTableSlot *tslot;
@@ -948,18 +998,34 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey)");
+
+ if(server_version >= 120000)
+ appendStringInfo(&cmd, ", a.attgenerated != ''");
+
+ appendStringInfo(&cmd,
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if(server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
+
+ if(!gencols_allowed)
+ {
+ /* Replication of generated cols is not supported. */
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+ }
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1039,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1072,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1084,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist_res = remotegenlist;
walrcv_clear_result(res);
/*
@@ -1037,7 +1106,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* 3) one of the subscribed publications is declared as TABLES IN SCHEMA
* that includes this relation
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
StringInfoData pub_names;
@@ -1123,10 +1192,13 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
+ bool gencol_copy_needed = false;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1207,31 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Check if the remote table has any generated columns that should be copied.
+ */
+ if (MySubscription->includegencols)
+ {
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if (remotegenlist[i])
+ {
+ gencol_copy_needed = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * Regular table with no row filter and copy of generated columns is
+ * not necessary.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL && !gencol_copy_needed)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1173,13 +1265,20 @@ copy_table(Relation rel)
* (SELECT ...), but we can't just do SELECT * because we need to not
* copy generated columns. For tables with any row filters, build a
* SELECT query with OR'ed row filters for COPY.
+ *
+ * We also need to use this same COPY (SELECT ...) syntax when
+ * 'include_generated_columns' is specified as true and the remote
+ * table has generated columns, because copy of generated columns is
+ * not supported by the normal COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_node(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1336,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 2bb96c1292..65197bede5 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,8 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7922dfd3cd..8c7381fbfc 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,7 +59,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index a2d9462395..9e35e678c1 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -172,6 +172,42 @@ ok( $node_publisher->poll_query_until(
'postgres', 'SELECT count(*) = 0 FROM pg_replication_slots'),
'DROP SUBSCRIPTION during error can clean up the slots on the publisher');
+# clean up
+$node_publisher->safe_psql('postgres', "DROP TABLE tab_rep");
+$node_subscriber->safe_psql('postgres', "DROP TABLE tab_rep");
+
+#
+# TEST CASE:
+#
+# When a subscription table has a column missing that was specified on
+# the publication table.
+#
+
+# setup structure with existing data on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_rep (a int, b int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_rep VALUES (1, 1), (2, 2), (3, 3)");
+
+# add table on subscriber; note column 'b' is missing
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_rep (a int)");
+
+my $offset = -s $node_subscriber->logfile;
+
+# create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+);
+
+# check for missing column error
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_rep" is missing replicated column: "b"/,
+ $offset);
+
+# clean up
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub");
+$node_subscriber->safe_psql('postgres', "DROP TABLE tab_rep");
+$node_publisher->safe_psql('postgres', "DROP TABLE tab_rep");
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index fe32987e6a..181462861a 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -16,6 +16,8 @@ $node_publisher->start;
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 10");
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -43,6 +45,43 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+# tab4:
+# publisher-side tab4 has generated cols 'b' and 'c' but
+# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
+# where columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab4 (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab5:
+# publisher-side tab5 has non-generated col 'b' but
+# subscriber-side tab5 has generated col 'b'
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# tab6:
+# tables for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab7:
+# publisher-side tab7 has generated col 'b' but
+# subscriber-side tab7 do not have col 'b'
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab7 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab7 (a int)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -51,6 +90,14 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab5 (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab7 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR TABLE tab1");
@@ -58,15 +105,24 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub2 FOR TABLE tab2");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub4 FOR TABLE tab4");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub5 FOR TABLE tab5");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub7 FOR TABLE tab7");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr' PUBLICATION pub4 WITH (include_generated_columns = true)"
);
# Wait for initial sync of all subscriptions
@@ -78,10 +134,24 @@ is( $result, qq(1|22
3|66), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'generated column initial sync');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(), 'unsubscribed table initial data');
# data to replicate
@@ -109,7 +179,10 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
@@ -127,11 +200,102 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub3');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
-is( $result, qq(4|24
+is( $result, qq(1|21
+2|22
+3|23
+4|24
5|25),
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
+#
+# TEST tab4: the publisher-side cols 'b' and 'c' are generated and subscriber-side
+# col 'b' is not generated and col 'c' is generated. So confirmed that the different
+# order of columns on subscriber-side replicate data to correct columns.
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub4');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated columns with different order on subscriber');
+
+#
+# TEST tab5: publisher-side col 'b' is not-generated and subscriber-side col 'b'
+# is generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
+# The subscription sub5 is created here, instead of earlier with the other subscriptions,
+# because sub5 will cause the tablesync worker to restart repetitively.
+#
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub5 CONNECTION '$publisher_connstr' PUBLICATION pub5 WITH (include_generated_columns = true)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab5" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub5");
+
+#
+# TEST tab6: After ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub4 ADD TABLE tab6");
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub4 REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('sub4');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'add new table to existing publication');
+
+#
+# TEST tab6: Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (4), (5)");
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
+#
+# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
+# 'include_generated_column' is 'true' so confirmed that col 'b' IS NOT replicated and
+# it will throw an error. The subscription sub7 is created here, instead of earlier with the
+# other subscriptions, because sub7 will cause the tablesync worker to restart repetitively.
+#
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7 with (include_generated_columns = true)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab7" is missing replicated column: "b"/,
+ $offset);
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub7");
+
+#
+# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
+# 'include_generated_column' is 'false' so confirmed that col 'b' IS NOT replicated.
+#
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7"
+);
+$node_publisher->wait_for_catchup('sub7');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a FROM tab7");
+is( $result, qq(1
+2
+3), 'missing generated column');
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
--
2.34.1
[application/octet-stream] v19-0004-Improve-include-generated-column-option-handling.patch (15.9K, ../../CAHv8Rj+R0cj=z1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A@mail.gmail.com/3-v19-0004-Improve-include-generated-column-option-handling.patch)
download | inline diff:
From b805362a5616ada8f7ce004b244d83267841f7fa Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Tue, 16 Jul 2024 15:24:52 +0530
Subject: [PATCH v19 4/4] Improve include generated column option handling by
using bms
Improve include generated column option handling by using bms.
---
src/backend/replication/logical/proto.c | 72 +++-------------
src/backend/replication/pgoutput/pgoutput.c | 94 ++++++++++++++-------
src/include/replication/logicalproto.h | 12 +--
src/test/subscription/t/031_column_list.pl | 2 +-
4 files changed, 80 insertions(+), 100 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index cad1b76e7a..6b085e555c 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,12 +30,10 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -414,8 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -427,8 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -461,8 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -483,13 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -539,7 +532,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -559,8 +552,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
/*
@@ -676,7 +668,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
char *relname;
@@ -698,7 +690,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns, include_generated_columns);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -775,8 +767,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -793,15 +784,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -823,15 +805,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -950,8 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
- bool include_generated_columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -969,15 +941,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -999,15 +962,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index a256ab7262..5ab1235c75 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,8 +86,7 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -165,8 +164,10 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns included in the publication, or NULL if all columns are
- * included implicitly. Note that the attnums in this bitmap are not
+ * Columns should be publicated, or NULL if all columns are included
+ * implicitly. This bitmap only considers the column list of the
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* publication and include_generated_columns option: other reasons should
* be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
@@ -746,13 +747,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -766,7 +765,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -786,15 +785,6 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -808,7 +798,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -1034,6 +1024,36 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Prepare new column list bitmap. This includes all the columns of the table.
+ */
+static Bitmapset *
+prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
+ TupleDesc desc)
+{
+ Bitmapset *cols = NULL;
+ MemoryContext oldcxt = NULL;
+
+ pgoutput_ensure_entry_cxt(data, entry);
+ oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped */
+ if (att->attisdropped)
+ continue;
+
+ /* Iterate the cols until generated columns are found. */
+ cols = bms_add_member(cols, i + 1);
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+
+ return cols;
+}
+
/*
* Initialize the column list.
*/
@@ -1072,7 +1092,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
* if there are no column lists (even if other publications have a
* list).
*/
- if (!pub->alltables)
+ if (!pub->alltables || !data->include_generated_columns)
{
bool pub_no_list = true;
@@ -1093,9 +1113,10 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
+ }
/* Build the column list bitmap in the per-entry context. */
- if (!pub_no_list) /* when not null */
+ if (!pub_no_list || !data->include_generated_columns) /* when not null */
{
int i;
int nliveatts = 0;
@@ -1103,19 +1124,31 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
pgoutput_ensure_entry_cxt(data, entry);
- cols = pub_collist_to_bitmapset(cols, cfdatum,
+ if (!pub_no_list)
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
entry->entry_cxt);
+ else
+ cols = prepare_all_columns_bms(data, entry, desc);
/* Get the number of live attributes. */
for (i = 0; i < desc->natts; i++)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
+ /* Skip if the attribute is dropped */
if (att->attisdropped)
continue;
-
- if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
+ /*
+ * If column list contain generated column it will not replicate
+ * the table to the subscriber port.
+ */
+ if (att->attgenerated &&
+ att->attgenerated != ATTRIBUTE_GENERATED_STORED &&
+ !data->include_generated_columns)
+ {
+ cols = bms_del_member(cols, i + 1);
+ continue;
+ }
nliveatts++;
}
@@ -1131,8 +1164,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
}
}
+ if (HeapTupleIsValid(cftuple))
ReleaseSysCache(cftuple);
- }
}
if (first)
@@ -1560,18 +1593,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns,
- data->include_generated_columns);
+ new_slot, data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
default:
Assert(false);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 34ec40b07e..b9a64d9c95 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,22 +225,19 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
TupleTableSlot *newslot, bool binary,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -251,8 +248,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns,
- bool include_generated_columns);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 3bb2301b43..60ad5751bc 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1247,7 +1247,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql(
'postgres', "SELECT * FROM test_mix_4 ORDER BY a"),
qq(1|2||
-3|4||),
+3|4||4),
'replication with multiple publications with the same column list');
# TEST: With a table included in multiple publications with different column
--
2.34.1
[application/octet-stream] v19-0001-Enable-support-for-include_generated_columns-opt.patch (89.6K, ../../CAHv8Rj+R0cj=z1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A@mail.gmail.com/4-v19-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 43358cf00ca133d7d032970ba709431cff0e945a Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Tue, 16 Jul 2024 13:53:01 +0530
Subject: [PATCH v19 1/4] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=postgres' PUBLICATION pub2 WITH
(include_generated_columns = true, copy_data = false);
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to
inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 43 +++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 72 +++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 451 insertions(+), 138 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index c5e11a6699..f7c57d47af 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..226c3641b9 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6524,8 +6535,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 16d83b3253..507c5ef9c1 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6c42c209d2..a762051732 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c0bda6269b..6fe2ff2ffa 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..6bc9f9d403 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -166,6 +167,8 @@ typedef struct RelationSyncEntry
/*
* Columns included in the publication, or NULL if all columns are
* included implicitly. Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
@@ -283,11 +286,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +401,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +746,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +766,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +783,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +802,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1105,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1551,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..e99f528e39 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5167,6 +5177,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..f7b8d59413 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..50c5911d23 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..9275b3a617 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 5c2f1ee517..2bb96c1292 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +376,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +388,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +401,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +417,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..7922dfd3cd 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..fe32987e6a 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,16 +28,46 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
+
+# tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+);
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +77,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is($result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is($result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,6 +98,40 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#
+# TEST tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
+#
+# Confirm that col 'b' is replicated.
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub2');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+);
+
+#
+# TEST tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
+#
+# Confirm that col 'b' is NOT replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub3');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.34.1
[application/octet-stream] v19-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (15.5K, ../../CAHv8Rj+R0cj=z1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A@mail.gmail.com/5-v19-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From 5db8adb8a3cb758b98f76bb040a7635838963437 Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Tue, 16 Jul 2024 14:48:14 +0530
Subject: [PATCH v19 3/4] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
.../expected/generated_columns.out | 1 +
.../test_decoding/sql/generated_columns.sql | 4 +-
contrib/test_decoding/test_decoding.c | 15 ++++++-
doc/src/sgml/protocol.sgml | 7 ++--
doc/src/sgml/ref/create_subscription.sgml | 4 +-
src/backend/catalog/pg_publication.c | 12 ++++++
src/backend/replication/logical/proto.c | 40 +++++++++++++++----
src/backend/replication/logical/tablesync.c | 8 +++-
src/backend/replication/pgoutput/pgoutput.c | 13 +++++-
src/test/subscription/t/011_generated.pl | 32 +++++++--------
10 files changed, 101 insertions(+), 35 deletions(-)
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index f3b26aa9e1..a79510705c 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -50,3 +50,4 @@ SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
stop
(1 row)
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 6d6d1d6564..997cdebc7e 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -19,4 +19,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..a847050f6e 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,8 +557,19 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
- continue;
+ if (attr->attgenerated)
+ {
+ /*
+ * Don't print generated columns when
+ * 'include_generated_columns' is false.
+ */
+ if (!include_generated_columns)
+ continue;
+
+ /* Don't print generated columns unless they are STORED. */
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
/*
* Don't print system columns, oid will already have been printed if
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 226c3641b9..06554fb2af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3310,9 +3310,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..1809e140ea 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -520,6 +521,7 @@ publication_translate_columns(Relation targetrel, List *columns,
{
char *colname = strVal(lfirst(lc));
AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+ Form_pg_attribute att;
if (attnum == InvalidAttrNumber)
ereport(ERROR,
@@ -533,6 +535,13 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ att = TupleDescAttr(tupdesc, attnum - 1);
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("generated column \"%s\" is not STORED so cannot be used in a publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1228,6 +1237,9 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
attnums[nattnums++] = att->attnum;
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e694baca0a..cad1b76e7a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,8 +793,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -817,8 +823,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -957,8 +969,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -981,8 +999,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 935be7f934..b1407cc97d 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1014,7 +1014,13 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
{
bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
- if(!gencols_allowed)
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ /* TODO: use ATTRIBUTE_GENERATED_VIRTUAL*/
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
{
/* Replication of generated cols is not supported. */
appendStringInfo(&cmd, " AND a.attgenerated = ''");
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6bc9f9d403..a256ab7262 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -786,8 +786,14 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -1108,6 +1114,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 181462861a..8c18682b4c 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -31,23 +31,23 @@ $node_subscriber->safe_psql('postgres',
);
# tab2:
-# publisher-side tab2 has generated col 'b'.
+# publisher-side tab2 has stored generated col 'b'.
# subscriber-side tab2 has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
# tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# publisher-side tab3 has stored generated col 'b' but
+# subscriber-side tab3 has DIFFERENT COMPUTATION stored generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
# tab4:
-# publisher-side tab4 has generated cols 'b' and 'c' but
-# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
+# publisher-side tab4 has stored generated cols 'b' and 'c' but
+# subscriber-side tab4 has non-generated col 'b', and stored generated-col 'c'
# where columns on publisher/subscriber are in a different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -58,7 +58,7 @@ $node_subscriber->safe_psql('postgres',
# tab5:
# publisher-side tab5 has non-generated col 'b' but
-# subscriber-side tab5 has generated col 'b'
+# subscriber-side tab5 has stored generated col 'b'
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
@@ -73,7 +73,7 @@ $node_subscriber->safe_psql('postgres',
);
# tab7:
-# publisher-side tab7 has generated col 'b' but
+# publisher-side tab7 has stored generated col 'b' but
# subscriber-side tab7 do not have col 'b'
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab7 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -170,7 +170,7 @@ is( $result, qq(1|22|
#
# TEST tab2:
-# publisher-side tab2 has generated col 'b'.
+# publisher-side tab2 has stored generated col 'b'.
# subscriber-side tab2 has non-generated col 'b'.
#
# Confirm that col 'b' is replicated.
@@ -189,7 +189,7 @@ is( $result, qq(1|2
#
# TEST tab3:
-# publisher-side tab3 has generated col 'b'.
+# publisher-side tab3 has stored generated col 'b'.
# subscriber-side tab3 has generated col 'b', using a different computation.
#
# Confirm that col 'b' is NOT replicated. We can know this because the result
@@ -209,8 +209,8 @@ is( $result, qq(1|21
);
#
-# TEST tab4: the publisher-side cols 'b' and 'c' are generated and subscriber-side
-# col 'b' is not generated and col 'c' is generated. So confirmed that the different
+# TEST tab4: the publisher-side cols 'b' and 'c' are stored generated and subscriber-side
+# col 'b' is not generated and col 'c' is stored generated. So confirmed that the different
# order of columns on subscriber-side replicate data to correct columns.
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
@@ -226,7 +226,7 @@ is( $result, qq(1|2|22
#
# TEST tab5: publisher-side col 'b' is not-generated and subscriber-side col 'b'
-# is generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
+# is stored generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
# The subscription sub5 is created here, instead of earlier with the other subscriptions,
# because sub5 will cause the tablesync worker to restart repetitively.
#
@@ -254,8 +254,8 @@ is( $result, qq(1|2|22
3|6|66), 'add new table to existing publication');
#
-# TEST tab6: Drop the generated column's expression on subscriber side.
-# This changes the generated column into a non-generated column.
+# TEST tab6: Drop the stored generated column's expression on subscriber side.
+# This changes the stored generated column into a non-generated column.
#
$node_subscriber->safe_psql('postgres',
"ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
@@ -270,7 +270,7 @@ is( $result, qq(1|2|22
5|10|10), 'after drop generated column expression');
#
-# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
+# TEST tab7: publisher-side col 'b' is stored generated and subscriber-side do not have col 'b' and
# 'include_generated_column' is 'true' so confirmed that col 'b' IS NOT replicated and
# it will throw an error. The subscription sub7 is created here, instead of earlier with the
# other subscriptions, because sub7 will cause the tablesync worker to restart repetitively.
@@ -284,7 +284,7 @@ $node_subscriber->wait_for_log(
$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub7");
#
-# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
+# TEST tab7: publisher-side col 'b' is stored generated and subscriber-side do not have col 'b' and
# 'include_generated_column' is 'false' so confirmed that col 'b' IS NOT replicated.
#
$node_subscriber->safe_psql('postgres',
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-18 05:17 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
2 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-18 05:17 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham, here are my review comments for patch v19-0001.
======
src/backend/replication/pgoutput/pgoutput.c
1.
/*
* Columns included in the publication, or NULL if all columns are
* included implicitly. Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
You replied [1] "The attached Patches contain all the suggested
changes." but as I previously commented [2, #1], since there is no
change to the interpretation of the 'columns' BMS caused by this
patch, then I expected this comment would be unchanged (i.e. same as
HEAD code). But this fix was missed in v19-0001.
OTOH, if you do think there was a reason to change the comment then
the above is still not good because "are not publication and
include_generated_columns option" wording doesn't make sense.
======
src/test/subscription/t/011_generated.pl
Observation -- I added (in nitpicks diffs) some more comments for
'tab1' (to make all comments consistent with the new tests added). But
when I was doing that I observed that tab1 and tab3 test scenarios are
very similar. It seems only the subscription parameter is not
specified (so 'include_generated_cols' default wll be tested). IIRC
the default for that parameter is "false", so tab1 is not really
testing that properly -- e.g. I thought maybe to test the default
parameter it's better the subscriber-side 'b' should be not-generated?
But doing that would make 'tab1' the same as 'tab2'. Anyway, something
seems amiss -- it seems either something is not tested or is duplicate
tested. Please revisit what the tab1 test intention was and make sure
we are doing the right thing for it...
======
99.
The attached nitpicks diff patch has some tweaked comments.
======
[1] https://www.postgresql.org/message-id/CAHv8Rj%2BR0cj%3Dz1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A%40mail...
[2] https://www.postgresql.org/message-id/CAHut%2BPtVfrbx0jb42LCmS%3D-LcMTtWxm%2BvhaoArkjg7Z0mvuXbg%40ma...
Kind Regards,
Peter Smith.
Fujitsu Australia.
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 50c5911..e066426 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,8 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
- bool subincludegencols; /* True if generated columns must be
- * published */
+ bool subincludegencols; /* True if generated columns should
+ * be published */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index fe32987..d13d0a0 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -20,24 +20,28 @@ $node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+#
+# tab1:
+# Publisher-side tab1 has generated col 'b'.
+# Subscriber-side tab1 has generated col 'b', using a different computation,
+# and also an additional column 'c'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
-
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
# tab2:
-# publisher-side tab2 has generated col 'b'.
-# subscriber-side tab2 has non-generated col 'b'.
+# Publisher-side tab2 has generated col 'b'.
+# Subscriber-side tab2 has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
# tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# Publisher-side tab3 has generated col 'b'.
+# Subscriber-side tab3 has generated col 'b', using a different computation.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
@@ -85,12 +89,19 @@ is($result, qq(), 'generated columns initial sync');
# data to replicate
+#
+# TEST tab1:
+# Publisher-side tab1 has generated col 'b'.
+# Subscriber-side tab1 has generated col 'b', using a different computation,
+# and also an additional column 'c'.
+#
+# Confirm that col 'b' is not replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+#
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
-
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-
$node_publisher->wait_for_catchup('sub1');
-
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
2|44|
@@ -100,8 +111,8 @@ is( $result, qq(1|22|
#
# TEST tab2:
-# publisher-side tab2 has generated col 'b'.
-# subscriber-side tab2 has non-generated col 'b'.
+# Publisher-side tab2 has generated col 'b'.
+# Subscriber-side tab2 has non-generated col 'b'.
#
# Confirm that col 'b' is replicated.
#
@@ -111,15 +122,15 @@ $result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
5|10),
- 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
);
#
# TEST tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# Publisher-side tab3 has generated col 'b'.
+# Subscriber-side tab3 has generated col 'b', using a different computation.
#
-# Confirm that col 'b' is NOT replicated. We can know this because the result
+# Confirm that col 'b' is not replicated. We can know this because the result
# value is the subscriber-side computation (which is different from the
# publisher-side computation for this column).
#
@@ -129,7 +140,7 @@ $result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
is( $result, qq(4|24
5|25),
- 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+ 'confirm generated columns are not replicated when the subscriber-side column is also generated'
);
# try it with a subscriber-side trigger
Attachments:
[text/plain] PS_NITPICKS_20240718_GENCOLS_V190001.txt (4.6K, ../../CAHut+PtLVgYJO96+YHGm=n8hByLcwj-+fVTXY+QOi_hGT74MsA@mail.gmail.com/2-PS_NITPICKS_20240718_GENCOLS_V190001.txt)
download | inline diff:
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 50c5911..e066426 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,8 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
- bool subincludegencols; /* True if generated columns must be
- * published */
+ bool subincludegencols; /* True if generated columns should
+ * be published */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index fe32987..d13d0a0 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -20,24 +20,28 @@ $node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+#
+# tab1:
+# Publisher-side tab1 has generated col 'b'.
+# Subscriber-side tab1 has generated col 'b', using a different computation,
+# and also an additional column 'c'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
-
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
# tab2:
-# publisher-side tab2 has generated col 'b'.
-# subscriber-side tab2 has non-generated col 'b'.
+# Publisher-side tab2 has generated col 'b'.
+# Subscriber-side tab2 has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
# tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# Publisher-side tab3 has generated col 'b'.
+# Subscriber-side tab3 has generated col 'b', using a different computation.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
@@ -85,12 +89,19 @@ is($result, qq(), 'generated columns initial sync');
# data to replicate
+#
+# TEST tab1:
+# Publisher-side tab1 has generated col 'b'.
+# Subscriber-side tab1 has generated col 'b', using a different computation,
+# and also an additional column 'c'.
+#
+# Confirm that col 'b' is not replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+#
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
-
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-
$node_publisher->wait_for_catchup('sub1');
-
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
2|44|
@@ -100,8 +111,8 @@ is( $result, qq(1|22|
#
# TEST tab2:
-# publisher-side tab2 has generated col 'b'.
-# subscriber-side tab2 has non-generated col 'b'.
+# Publisher-side tab2 has generated col 'b'.
+# Subscriber-side tab2 has non-generated col 'b'.
#
# Confirm that col 'b' is replicated.
#
@@ -111,15 +122,15 @@ $result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
is( $result, qq(4|8
5|10),
- 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
);
#
# TEST tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# Publisher-side tab3 has generated col 'b'.
+# Subscriber-side tab3 has generated col 'b', using a different computation.
#
-# Confirm that col 'b' is NOT replicated. We can know this because the result
+# Confirm that col 'b' is not replicated. We can know this because the result
# value is the subscriber-side computation (which is different from the
# publisher-side computation for this column).
#
@@ -129,7 +140,7 @@ $result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
is( $result, qq(4|24
5|25),
- 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+ 'confirm generated columns are not replicated when the subscriber-side column is also generated'
);
# try it with a subscriber-side trigger
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-18 08:24 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
2 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-18 08:24 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, here are some review comments for v19-0002
======
src/backend/replication/logical/tablesync.c
make_copy_attnamelist:
nitpick - tweak function comment
nitpick - tweak other comments
~~~
fetch_remote_table_info:
nitpick - add space after "if"
nitpick - removed a comment because logic is self-evident from the variable name
======
src/test/subscription/t/004_sync.pl
1.
This new test is not related to generated columns. IIRC, this is just
some test that we discovered missing during review of this thread. As
such, I think this change can be posted/patched separately from this
thread.
======
src/test/subscription/t/011_generated.pl
nitpick - change some comment wording to be more consistent with patch 0001.
======
99.
Please see the nitpicks diff attachment which implements any nitpicks
mentioned above.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 935be7f..2e90d42 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -693,8 +693,8 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping. Do not
- * include generated columns of the subscription table in the column list.
+ * Create list of columns for COPY based on logical relation mapping.
+ * Exclude columns that are subscription table generated columns.
*/
static List *
make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
@@ -707,7 +707,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
/*
- * This loop checks for generated columns on subscription table.
+ * This loop checks for generated columns of the subscription table.
*/
for (int i = 0; i < desc->natts; i++)
{
@@ -1010,15 +1010,12 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
" WHERE a.attnum > 0::pg_catalog.int2"
" AND NOT a.attisdropped", lrel->remoteid);
- if(server_version >= 120000)
+ if (server_version >= 120000)
{
bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
- if(!gencols_allowed)
- {
- /* Replication of generated cols is not supported. */
+ if (!gencols_allowed)
appendStringInfo(&cmd, " AND a.attgenerated = ''");
- }
}
appendStringInfo(&cmd,
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 1814628..4537c6c 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -46,9 +46,9 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
# tab4:
-# publisher-side tab4 has generated cols 'b' and 'c' but
-# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
-# where columns on publisher/subscriber are in a different order
+# Publisher-side tab4 has generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Columns on publisher/subscriber are in a different order.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -57,14 +57,14 @@ $node_subscriber->safe_psql('postgres',
);
# tab5:
-# publisher-side tab5 has non-generated col 'b' but
-# subscriber-side tab5 has generated col 'b'
+# Publisher-side tab5 has non-generated col 'b'.
+# Subscriber-side tab5 has generated col 'b'.
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
# tab6:
-# tables for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+# Tables for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -73,8 +73,8 @@ $node_subscriber->safe_psql('postgres',
);
# tab7:
-# publisher-side tab7 has generated col 'b' but
-# subscriber-side tab7 do not have col 'b'
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab7 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -209,9 +209,12 @@ is( $result, qq(1|21
);
#
-# TEST tab4: the publisher-side cols 'b' and 'c' are generated and subscriber-side
-# col 'b' is not generated and col 'c' is generated. So confirmed that the different
-# order of columns on subscriber-side replicate data to correct columns.
+# TEST tab4:
+# Publisher-side tab4 has generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Columns on publisher/subscriber are in a different order.
+#
+# Confirm despite the different order columns, they still replicate correctly.
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub4');
@@ -225,10 +228,15 @@ is( $result, qq(1|2|22
5|10|110), 'replicate generated columns with different order on subscriber');
#
-# TEST tab5: publisher-side col 'b' is not-generated and subscriber-side col 'b'
-# is generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
-# The subscription sub5 is created here, instead of earlier with the other subscriptions,
-# because sub5 will cause the tablesync worker to restart repetitively.
+# TEST tab5:
+# Publisher-side tab5 has non-generated col 'b'.
+# Subscriber-side tab5 has generated col 'b'.
+#
+# Confirm that col 'b' is not replicated and it will throw an error.
+#
+# Note that subscription sub5 is created here, instead of earlier with the
+# other subscriptions, because sub5 will cause the tablesync worker to restart
+# repetitively.
#
my $offset = -s $node_subscriber->logfile;
$node_subscriber->safe_psql('postgres',
@@ -254,9 +262,13 @@ is( $result, qq(1|2|22
3|6|66), 'add new table to existing publication');
#
-# TEST tab6: Drop the generated column's expression on subscriber side.
+# TEST tab6:
+# Drop the generated column's expression on subscriber side.
# This changes the generated column into a non-generated column.
#
+# Confirm that replication happens after the drop expression, because now we
+# are replicating from a generated column to a non-generated column.
+#
$node_subscriber->safe_psql('postgres',
"ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
$node_publisher->safe_psql('postgres',
@@ -270,10 +282,16 @@ is( $result, qq(1|2|22
5|10|10), 'after drop generated column expression');
#
-# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
-# 'include_generated_column' is 'true' so confirmed that col 'b' IS NOT replicated and
-# it will throw an error. The subscription sub7 is created here, instead of earlier with the
-# other subscriptions, because sub7 will cause the tablesync worker to restart repetitively.
+# TEST tab7, false
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
+# 'include_generated_columns' is true.
+#
+# Confirm that attempted replication of col 'b' will throw an error.
+#
+# Note the subscription sub7 is created here, instead of earlier with the
+# other subscriptions, because sub7 will cause the tablesync worker to restart
+# repetitively.
#
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7 with (include_generated_columns = true)"
@@ -284,8 +302,12 @@ $node_subscriber->wait_for_log(
$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub7");
#
-# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
-# 'include_generated_column' is 'false' so confirmed that col 'b' IS NOT replicated.
+# TEST tab7:
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
+# 'include_generated_columns' is default (false).
+#
+# Confirm that col 'b' is not replicated, and no error occurs.
#
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7"
Attachments:
[text/plain] PS_NITPICKS_20240718_GENCOLS_v190002.txt (7.4K, ../../CAHut+Pt5FqV7J9GnnWFRNW_Z1KOMMAZXNTRcRNdtFrfMBz_GLA@mail.gmail.com/2-PS_NITPICKS_20240718_GENCOLS_v190002.txt)
download | inline diff:
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 935be7f..2e90d42 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -693,8 +693,8 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping. Do not
- * include generated columns of the subscription table in the column list.
+ * Create list of columns for COPY based on logical relation mapping.
+ * Exclude columns that are subscription table generated columns.
*/
static List *
make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
@@ -707,7 +707,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
/*
- * This loop checks for generated columns on subscription table.
+ * This loop checks for generated columns of the subscription table.
*/
for (int i = 0; i < desc->natts; i++)
{
@@ -1010,15 +1010,12 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
" WHERE a.attnum > 0::pg_catalog.int2"
" AND NOT a.attisdropped", lrel->remoteid);
- if(server_version >= 120000)
+ if (server_version >= 120000)
{
bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
- if(!gencols_allowed)
- {
- /* Replication of generated cols is not supported. */
+ if (!gencols_allowed)
appendStringInfo(&cmd, " AND a.attgenerated = ''");
- }
}
appendStringInfo(&cmd,
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 1814628..4537c6c 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -46,9 +46,9 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
# tab4:
-# publisher-side tab4 has generated cols 'b' and 'c' but
-# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
-# where columns on publisher/subscriber are in a different order
+# Publisher-side tab4 has generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Columns on publisher/subscriber are in a different order.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -57,14 +57,14 @@ $node_subscriber->safe_psql('postgres',
);
# tab5:
-# publisher-side tab5 has non-generated col 'b' but
-# subscriber-side tab5 has generated col 'b'
+# Publisher-side tab5 has non-generated col 'b'.
+# Subscriber-side tab5 has generated col 'b'.
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
# tab6:
-# tables for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+# Tables for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -73,8 +73,8 @@ $node_subscriber->safe_psql('postgres',
);
# tab7:
-# publisher-side tab7 has generated col 'b' but
-# subscriber-side tab7 do not have col 'b'
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab7 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -209,9 +209,12 @@ is( $result, qq(1|21
);
#
-# TEST tab4: the publisher-side cols 'b' and 'c' are generated and subscriber-side
-# col 'b' is not generated and col 'c' is generated. So confirmed that the different
-# order of columns on subscriber-side replicate data to correct columns.
+# TEST tab4:
+# Publisher-side tab4 has generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Columns on publisher/subscriber are in a different order.
+#
+# Confirm despite the different order columns, they still replicate correctly.
#
$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub4');
@@ -225,10 +228,15 @@ is( $result, qq(1|2|22
5|10|110), 'replicate generated columns with different order on subscriber');
#
-# TEST tab5: publisher-side col 'b' is not-generated and subscriber-side col 'b'
-# is generated, so confirmed that col 'b' IS NOT replicated and it will throw an error.
-# The subscription sub5 is created here, instead of earlier with the other subscriptions,
-# because sub5 will cause the tablesync worker to restart repetitively.
+# TEST tab5:
+# Publisher-side tab5 has non-generated col 'b'.
+# Subscriber-side tab5 has generated col 'b'.
+#
+# Confirm that col 'b' is not replicated and it will throw an error.
+#
+# Note that subscription sub5 is created here, instead of earlier with the
+# other subscriptions, because sub5 will cause the tablesync worker to restart
+# repetitively.
#
my $offset = -s $node_subscriber->logfile;
$node_subscriber->safe_psql('postgres',
@@ -254,9 +262,13 @@ is( $result, qq(1|2|22
3|6|66), 'add new table to existing publication');
#
-# TEST tab6: Drop the generated column's expression on subscriber side.
+# TEST tab6:
+# Drop the generated column's expression on subscriber side.
# This changes the generated column into a non-generated column.
#
+# Confirm that replication happens after the drop expression, because now we
+# are replicating from a generated column to a non-generated column.
+#
$node_subscriber->safe_psql('postgres',
"ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
$node_publisher->safe_psql('postgres',
@@ -270,10 +282,16 @@ is( $result, qq(1|2|22
5|10|10), 'after drop generated column expression');
#
-# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
-# 'include_generated_column' is 'true' so confirmed that col 'b' IS NOT replicated and
-# it will throw an error. The subscription sub7 is created here, instead of earlier with the
-# other subscriptions, because sub7 will cause the tablesync worker to restart repetitively.
+# TEST tab7, false
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
+# 'include_generated_columns' is true.
+#
+# Confirm that attempted replication of col 'b' will throw an error.
+#
+# Note the subscription sub7 is created here, instead of earlier with the
+# other subscriptions, because sub7 will cause the tablesync worker to restart
+# repetitively.
#
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7 with (include_generated_columns = true)"
@@ -284,8 +302,12 @@ $node_subscriber->wait_for_log(
$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub7");
#
-# TEST tab7: publisher-side col 'b' is generated and subscriber-side do not have col 'b' and
-# 'include_generated_column' is 'false' so confirmed that col 'b' IS NOT replicated.
+# TEST tab7:
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
+# 'include_generated_columns' is default (false).
+#
+# Confirm that col 'b' is not replicated, and no error occurs.
#
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7"
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-18 23:29 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
2 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-18 23:29 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, here are some review comments for patch v19-0003
======
src/backend/catalog/pg_publication.c
1.
/*
* Translate a list of column names to an array of attribute numbers
* and a Bitmapset with them; verify that each attribute is appropriate
* to have in a publication column list (no system or generated attributes,
* no duplicates). Additional checks with replica identity are done later;
* see pub_collist_contains_invalid_column.
*
* Note that the attribute numbers are *not* offset by
* FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
* is okay.
*/
static void
publication_translate_columns(Relation targetrel, List *columns,
int *natts, AttrNumber **attrs)
~
I though the above comment ought to change: /or generated
attributes/or virtual generated attributes/
IIRC this was already addressed back in v16, but somehow that fix has
been lost (???).
======
src/backend/replication/logical/tablesync.c
fetch_remote_table_info:
nitpick - missing end space in this comment /* TODO: use
ATTRIBUTE_GENERATED_VIRTUAL*/
======
2.
(in patch v19-0001)
+# tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
(here, in patch v19-0003)
# tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# publisher-side tab3 has stored generated col 'b' but
+# subscriber-side tab3 has DIFFERENT COMPUTATION stored generated col 'b'.
It has become difficult to review these TAP tests, particularly when
different patches are modifying the same comment. e.g. I post
suggestions to modify comments for patch 0001. Those get addressed OK,
only to vanish in subsequent patches like has happened in the above
example.
Really this patch 0003 was only supposed to add the word "stored", not
revert the entire comment to something from an earlier version. Please
take care that all comment changes are carried forward correctly from
one patch to the next.
======
Kind Regards,
Peter Smith.
Fujitsu Australia.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-19 06:01 Shlok Kyal <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Shlok Kyal @ 2024-07-19 06:01 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, 18 Jul 2024 at 13:55, Peter Smith <[email protected]> wrote:
>
> Hi, here are some review comments for v19-0002
> ======
> src/test/subscription/t/004_sync.pl
>
> 1.
> This new test is not related to generated columns. IIRC, this is just
> some test that we discovered missing during review of this thread. As
> such, I think this change can be posted/patched separately from this
> thread.
>
I have removed the test for this thread.
I have also addressed the remaining comments for v19-0002 patch.
Please find the latest patches.
v20-0001 - not modified
v20-0002 - Addressed the comments
v20-0003 - Addressed the comments
v20-0004 - Not modified
Thanks and Regards,
Shlok Kyal
Attachments:
[application/octet-stream] v20-0001-Enable-support-for-include_generated_columns-opt.patch (89.6K, ../../CANhcyEUzUurrX38HGvG30gV92YDz6WmnnwNRYMVY4tiga-8KZg@mail.gmail.com/2-v20-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From c25ccdb3f4e9623e4f193365f0284dc0a1bad67e Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Tue, 16 Jul 2024 13:53:01 +0530
Subject: [PATCH v20 1/4] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput plugin
or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=postgres' PUBLICATION pub2 WITH
(include_generated_columns = true, copy_data = false);
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to
inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 43 +++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 72 +++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 451 insertions(+), 138 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1b27d0a547..226c3641b9 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3306,6 +3306,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6524,8 +6535,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 16d83b3253..507c5ef9c1 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -99,6 +100,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -161,6 +163,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -366,6 +370,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -446,6 +459,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -603,7 +630,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -723,6 +751,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6c42c209d2..a762051732 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c0bda6269b..6fe2ff2ffa 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4379,6 +4379,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..6bc9f9d403 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -166,6 +167,8 @@ typedef struct RelationSyncEntry
/*
* Columns included in the publication, or NULL if all columns are
* included implicitly. Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
@@ -283,11 +286,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +401,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +746,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +766,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +783,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +802,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1105,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1551,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..e99f528e39 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5167,6 +5177,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..f7b8d59413 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..ade6a34eeb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..50c5911d23 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns must be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 12f71fa99b..9275b3a617 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 5c2f1ee517..2bb96c1292 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +376,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +388,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +401,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +417,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..7922dfd3cd 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..fe32987e6a 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -28,16 +28,46 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
+
+# tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+);
# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;
@@ -47,6 +77,12 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
+is($result, qq(), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is($result, qq(), 'generated columns initial sync');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,6 +98,40 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#
+# TEST tab2:
+# publisher-side tab2 has generated col 'b'.
+# subscriber-side tab2 has non-generated col 'b'.
+#
+# Confirm that col 'b' is replicated.
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub2');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns ARE replicated when the subscriber-side column is not generated'
+);
+
+#
+# TEST tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.
+#
+# Confirm that col 'b' is NOT replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub3');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.34.1
[application/octet-stream] v20-0002-Support-replication-of-generated-column-during-i.patch (25.9K, ../../CANhcyEUzUurrX38HGvG30gV92YDz6WmnnwNRYMVY4tiga-8KZg@mail.gmail.com/3-v20-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From eda5b9e496170ad7bdad175800c97d9ae1d4c318 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Tue, 16 Jul 2024 11:11:50 +0530
Subject: [PATCH v20 2/4] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Summary:
when (include_generated_columns = true)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column: The
publisher generated column value is copied.
* publisher generated column => subscriber generated column: The
publisher generated column value is not copied. The subscriber
generated column will be filled with the subscriber-side computed or
default data.
when (include_generated_columns = false)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column:
Publisher generated column is not replicated. The subscriber column
will be filled with the subscriber-side default data.
* publisher generated column => subscriber generated column: Publisher
generated column is not replicated. The subscriber generated column
will be filed with the subscriber-side computed or default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 --
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/tablesync.c | 139 +++++++++++---
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 2 -
src/test/regress/sql/subscription.sql | 1 -
src/test/subscription/t/011_generated.pl | 198 +++++++++++++++++++-
8 files changed, 312 insertions(+), 51 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 507c5ef9c1..0847c174c1 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -459,20 +459,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..9de0b75330 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e03e761392..2e90d42bdc 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -693,20 +694,67 @@ process_syncing_tables(XLogRecPtr current_lsn)
/*
* Create list of columns for COPY based on logical relation mapping.
+ * Exclude columns that are subscription table generated columns.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *localgenlist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
+
+ /*
+ * This loop checks for generated columns of the subscription table.
+ */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int remote_attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (remote_attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if (!remotegenlist[remote_attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'localgenlist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ localgenlist[remote_attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY, excluding columns that are
+ * subscription table generated columns.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if (!localgenlist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
+ pfree(localgenlist);
return attnamelist;
}
@@ -791,19 +839,21 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
+ int server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
lrel->nspname = nspname;
lrel->relname = relname;
@@ -851,7 +901,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
WalRcvExecResult *pubres;
TupleTableSlot *tslot;
@@ -948,18 +998,31 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey)");
+
+ if(server_version >= 120000)
+ appendStringInfo(&cmd, ", a.attgenerated != ''");
+
+ appendStringInfo(&cmd,
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if (server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
+
+ if (!gencols_allowed)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1036,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1069,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1081,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist_res = remotegenlist;
walrcv_clear_result(res);
/*
@@ -1037,7 +1103,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* 3) one of the subscribed publications is declared as TABLES IN SCHEMA
* that includes this relation
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
StringInfoData pub_names;
@@ -1123,10 +1189,13 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
+ bool gencol_copy_needed = false;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1204,31 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Check if the remote table has any generated columns that should be copied.
+ */
+ if (MySubscription->includegencols)
+ {
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if (remotegenlist[i])
+ {
+ gencol_copy_needed = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * Regular table with no row filter and copy of generated columns is
+ * not necessary.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL && !gencol_copy_needed)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1173,13 +1262,20 @@ copy_table(Relation rel)
* (SELECT ...), but we can't just do SELECT * because we need to not
* copy generated columns. For tables with any row filters, build a
* SELECT query with OR'ed row filters for COPY.
+ *
+ * We also need to use this same COPY (SELECT ...) syntax when
+ * 'include_generated_columns' is specified as true and the remote
+ * table has generated columns, because copy of generated columns is
+ * not supported by the normal COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_node(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1333,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 2bb96c1292..65197bede5 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,8 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7922dfd3cd..8c7381fbfc 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,7 +59,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index fe32987e6a..4537c6c5da 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -16,6 +16,8 @@ $node_publisher->start;
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 10");
$node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -43,6 +45,43 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+# tab4:
+# Publisher-side tab4 has generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Columns on publisher/subscriber are in a different order.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab4 (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab5:
+# Publisher-side tab5 has non-generated col 'b'.
+# Subscriber-side tab5 has generated col 'b'.
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# tab6:
+# Tables for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab6 (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab7:
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab7 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab7 (a int)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
@@ -51,6 +90,14 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab5 (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab7 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR TABLE tab1");
@@ -58,15 +105,24 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub2 FOR TABLE tab2");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub4 FOR TABLE tab4");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub5 FOR TABLE tab5");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub7 FOR TABLE tab7");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (include_generated_columns = true)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (include_generated_columns = true)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr' PUBLICATION pub4 WITH (include_generated_columns = true)"
);
# Wait for initial sync of all subscriptions
@@ -78,10 +134,24 @@ is( $result, qq(1|22
3|66), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync');
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
-is($result, qq(), 'generated columns initial sync');
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'generated column initial sync');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(), 'unsubscribed table initial data');
# data to replicate
@@ -109,7 +179,10 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub2');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab2 ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm generated columns ARE replicated when the subscriber-side column is not generated'
);
@@ -127,11 +200,124 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
$node_publisher->wait_for_catchup('sub3');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3 ORDER BY a");
-is( $result, qq(4|24
+is( $result, qq(1|21
+2|22
+3|23
+4|24
5|25),
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
+#
+# TEST tab4:
+# Publisher-side tab4 has generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Columns on publisher/subscriber are in a different order.
+#
+# Confirm despite the different order columns, they still replicate correctly.
+#
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (4), (5)");
+$node_publisher->wait_for_catchup('sub4');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab4 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
+5|10|110), 'replicate generated columns with different order on subscriber');
+
+#
+# TEST tab5:
+# Publisher-side tab5 has non-generated col 'b'.
+# Subscriber-side tab5 has generated col 'b'.
+#
+# Confirm that col 'b' is not replicated and it will throw an error.
+#
+# Note that subscription sub5 is created here, instead of earlier with the
+# other subscriptions, because sub5 will cause the tablesync worker to restart
+# repetitively.
+#
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub5 CONNECTION '$publisher_connstr' PUBLICATION pub5 WITH (include_generated_columns = true)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab5" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub5");
+
+#
+# TEST tab6: After ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub4 ADD TABLE tab6");
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub4 REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('sub4');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'add new table to existing publication');
+
+#
+# TEST tab6:
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#
+# Confirm that replication happens after the drop expression, because now we
+# are replicating from a generated column to a non-generated column.
+#
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab6 (a) VALUES (4), (5)");
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM tab6 ORDER BY a");
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
+#
+# TEST tab7, false
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
+# 'include_generated_columns' is true.
+#
+# Confirm that attempted replication of col 'b' will throw an error.
+#
+# Note the subscription sub7 is created here, instead of earlier with the
+# other subscriptions, because sub7 will cause the tablesync worker to restart
+# repetitively.
+#
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7 with (include_generated_columns = true)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab7" is missing replicated column: "b"/,
+ $offset);
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub7");
+
+#
+# TEST tab7:
+# Publisher-side tab7 has generated col 'b'.
+# Subscriber-side tab7 does not have any col 'b'.
+# 'include_generated_columns' is default (false).
+#
+# Confirm that col 'b' is not replicated, and no error occurs.
+#
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub7 CONNECTION '$publisher_connstr' PUBLICATION pub7"
+);
+$node_publisher->wait_for_catchup('sub7');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a FROM tab7");
+is( $result, qq(1
+2
+3), 'missing generated column');
+
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
--
2.34.1
[application/octet-stream] v20-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (15.7K, ../../CANhcyEUzUurrX38HGvG30gV92YDz6WmnnwNRYMVY4tiga-8KZg@mail.gmail.com/4-v20-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From 4d7f8d96486600561d4be0afd96c8b92c63bf41a Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Fri, 19 Jul 2024 11:00:14 +0530
Subject: [PATCH v20 3/4] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
.../expected/generated_columns.out | 1 +
.../test_decoding/sql/generated_columns.sql | 4 +-
contrib/test_decoding/test_decoding.c | 15 ++++++-
doc/src/sgml/protocol.sgml | 7 ++--
doc/src/sgml/ref/create_subscription.sgml | 4 +-
src/backend/catalog/pg_publication.c | 18 +++++++--
src/backend/replication/logical/proto.c | 40 +++++++++++++++----
src/backend/replication/logical/tablesync.c | 8 +++-
src/backend/replication/pgoutput/pgoutput.c | 13 +++++-
src/test/subscription/t/011_generated.pl | 36 ++++++++---------
10 files changed, 106 insertions(+), 40 deletions(-)
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index f3b26aa9e1..a79510705c 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -50,3 +50,4 @@ SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
stop
(1 row)
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 6d6d1d6564..997cdebc7e 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -19,4 +19,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..a847050f6e 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,8 +557,19 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
- continue;
+ if (attr->attgenerated)
+ {
+ /*
+ * Don't print generated columns when
+ * 'include_generated_columns' is false.
+ */
+ if (!include_generated_columns)
+ continue;
+
+ /* Don't print generated columns unless they are STORED. */
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
/*
* Don't print system columns, oid will already have been printed if
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 226c3641b9..06554fb2af 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3310,9 +3310,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..71466b1583 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -490,9 +490,9 @@ compare_int16(const void *a, const void *b)
/*
* Translate a list of column names to an array of attribute numbers
* and a Bitmapset with them; verify that each attribute is appropriate
- * to have in a publication column list (no system or generated attributes,
- * no duplicates). Additional checks with replica identity are done later;
- * see pub_collist_contains_invalid_column.
+ * to have in a publication column list (no system or virtual generated
+ * attributes, no duplicates). Additional checks with replica identity
+ * are done later; see pub_collist_contains_invalid_column.
*
* Note that the attribute numbers are *not* offset by
* FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -520,6 +521,7 @@ publication_translate_columns(Relation targetrel, List *columns,
{
char *colname = strVal(lfirst(lc));
AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+ Form_pg_attribute att;
if (attnum == InvalidAttrNumber)
ereport(ERROR,
@@ -533,6 +535,13 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ att = TupleDescAttr(tupdesc, attnum - 1);
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("generated column \"%s\" is not STORED so cannot be used in a publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1228,6 +1237,9 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
attnums[nattnums++] = att->attnum;
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e694baca0a..cad1b76e7a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,8 +793,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -817,8 +823,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -957,8 +969,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -981,8 +999,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 2e90d42bdc..14a0aae416 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1014,7 +1014,13 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
{
bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
- if (!gencols_allowed)
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ /* TODO: use ATTRIBUTE_GENERATED_VIRTUAL */
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
appendStringInfo(&cmd, " AND a.attgenerated = ''");
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6bc9f9d403..a256ab7262 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -786,8 +786,14 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -1108,6 +1114,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 4537c6c5da..b7dbe6117f 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -31,23 +31,23 @@ $node_subscriber->safe_psql('postgres',
);
# tab2:
-# publisher-side tab2 has generated col 'b'.
+# publisher-side tab2 has stored generated col 'b'.
# subscriber-side tab2 has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab2 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab2 (a int, b int)");
# tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# publisher-side tab3 has stored generated col 'b'.
+# subscriber-side tab3 has stored generated col 'b', using a different computation.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab3 (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
# tab4:
-# Publisher-side tab4 has generated cols 'b' and 'c'.
-# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Publisher-side tab4 has stored generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and stored generated-col 'c'.
# Columns on publisher/subscriber are in a different order.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab4 (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -58,7 +58,7 @@ $node_subscriber->safe_psql('postgres',
# tab5:
# Publisher-side tab5 has non-generated col 'b'.
-# Subscriber-side tab5 has generated col 'b'.
+# Subscriber-side tab5 has stored generated col 'b'.
$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab5 (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
@@ -73,7 +73,7 @@ $node_subscriber->safe_psql('postgres',
);
# tab7:
-# Publisher-side tab7 has generated col 'b'.
+# Publisher-side tab7 has stored generated col 'b'.
# Subscriber-side tab7 does not have any col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab7 (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -170,7 +170,7 @@ is( $result, qq(1|22|
#
# TEST tab2:
-# publisher-side tab2 has generated col 'b'.
+# publisher-side tab2 has stored generated col 'b'.
# subscriber-side tab2 has non-generated col 'b'.
#
# Confirm that col 'b' is replicated.
@@ -189,8 +189,8 @@ is( $result, qq(1|2
#
# TEST tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# publisher-side tab3 has stored generated col 'b'.
+# subscriber-side tab3 has stored generated col 'b', using a different computation.
#
# Confirm that col 'b' is NOT replicated. We can know this because the result
# value is the subscriber-side computation (which is different from the
@@ -210,8 +210,8 @@ is( $result, qq(1|21
#
# TEST tab4:
-# Publisher-side tab4 has generated cols 'b' and 'c'.
-# Subscriber-side tab4 has non-generated col 'b', and generated-col 'c'.
+# Publisher-side tab4 has stored generated cols 'b' and 'c'.
+# Subscriber-side tab4 has non-generated col 'b', and stored generated-col 'c'.
# Columns on publisher/subscriber are in a different order.
#
# Confirm despite the different order columns, they still replicate correctly.
@@ -230,7 +230,7 @@ is( $result, qq(1|2|22
#
# TEST tab5:
# Publisher-side tab5 has non-generated col 'b'.
-# Subscriber-side tab5 has generated col 'b'.
+# Subscriber-side tab5 has stored generated col 'b'.
#
# Confirm that col 'b' is not replicated and it will throw an error.
#
@@ -263,11 +263,11 @@ is( $result, qq(1|2|22
#
# TEST tab6:
-# Drop the generated column's expression on subscriber side.
-# This changes the generated column into a non-generated column.
+# Drop the stored generated column's expression on subscriber side.
+# This changes the stored generated column into a non-generated column.
#
# Confirm that replication happens after the drop expression, because now we
-# are replicating from a generated column to a non-generated column.
+# are replicating from a stored generated column to a non-generated column.
#
$node_subscriber->safe_psql('postgres',
"ALTER TABLE tab6 ALTER COLUMN c DROP EXPRESSION");
@@ -283,7 +283,7 @@ is( $result, qq(1|2|22
#
# TEST tab7, false
-# Publisher-side tab7 has generated col 'b'.
+# Publisher-side tab7 has stored generated col 'b'.
# Subscriber-side tab7 does not have any col 'b'.
# 'include_generated_columns' is true.
#
@@ -303,7 +303,7 @@ $node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub7");
#
# TEST tab7:
-# Publisher-side tab7 has generated col 'b'.
+# Publisher-side tab7 has stored generated col 'b'.
# Subscriber-side tab7 does not have any col 'b'.
# 'include_generated_columns' is default (false).
#
--
2.34.1
[application/octet-stream] v20-0004-Improve-include-generated-column-option-handling.patch (15.9K, ../../CANhcyEUzUurrX38HGvG30gV92YDz6WmnnwNRYMVY4tiga-8KZg@mail.gmail.com/5-v20-0004-Improve-include-generated-column-option-handling.patch)
download | inline diff:
From cf3d5097d2dce1c0dd5d0499f84e6d840de97176 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Tue, 16 Jul 2024 15:24:52 +0530
Subject: [PATCH v20 4/4] Improve include generated column option handling by
using bms
Improve include generated column option handling by using bms.
---
src/backend/replication/logical/proto.c | 72 +++-------------
src/backend/replication/pgoutput/pgoutput.c | 94 ++++++++++++++-------
src/include/replication/logicalproto.h | 12 +--
src/test/subscription/t/031_column_list.pl | 2 +-
4 files changed, 80 insertions(+), 100 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index cad1b76e7a..6b085e555c 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,12 +30,10 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -414,8 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -427,8 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -461,8 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -483,13 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -539,7 +532,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -559,8 +552,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
/*
@@ -676,7 +668,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
char *relname;
@@ -698,7 +690,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns, include_generated_columns);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -775,8 +767,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -793,15 +784,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -823,15 +805,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -950,8 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
- bool include_generated_columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -969,15 +941,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -999,15 +962,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index a256ab7262..5ab1235c75 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,8 +86,7 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -165,8 +164,10 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns included in the publication, or NULL if all columns are
- * included implicitly. Note that the attnums in this bitmap are not
+ * Columns should be publicated, or NULL if all columns are included
+ * implicitly. This bitmap only considers the column list of the
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* publication and include_generated_columns option: other reasons should
* be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
@@ -746,13 +747,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -766,7 +765,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -786,15 +785,6 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -808,7 +798,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -1034,6 +1024,36 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Prepare new column list bitmap. This includes all the columns of the table.
+ */
+static Bitmapset *
+prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
+ TupleDesc desc)
+{
+ Bitmapset *cols = NULL;
+ MemoryContext oldcxt = NULL;
+
+ pgoutput_ensure_entry_cxt(data, entry);
+ oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped */
+ if (att->attisdropped)
+ continue;
+
+ /* Iterate the cols until generated columns are found. */
+ cols = bms_add_member(cols, i + 1);
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+
+ return cols;
+}
+
/*
* Initialize the column list.
*/
@@ -1072,7 +1092,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
* if there are no column lists (even if other publications have a
* list).
*/
- if (!pub->alltables)
+ if (!pub->alltables || !data->include_generated_columns)
{
bool pub_no_list = true;
@@ -1093,9 +1113,10 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
+ }
/* Build the column list bitmap in the per-entry context. */
- if (!pub_no_list) /* when not null */
+ if (!pub_no_list || !data->include_generated_columns) /* when not null */
{
int i;
int nliveatts = 0;
@@ -1103,19 +1124,31 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
pgoutput_ensure_entry_cxt(data, entry);
- cols = pub_collist_to_bitmapset(cols, cfdatum,
+ if (!pub_no_list)
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
entry->entry_cxt);
+ else
+ cols = prepare_all_columns_bms(data, entry, desc);
/* Get the number of live attributes. */
for (i = 0; i < desc->natts; i++)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
+ /* Skip if the attribute is dropped */
if (att->attisdropped)
continue;
-
- if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
+ /*
+ * If column list contain generated column it will not replicate
+ * the table to the subscriber port.
+ */
+ if (att->attgenerated &&
+ att->attgenerated != ATTRIBUTE_GENERATED_STORED &&
+ !data->include_generated_columns)
+ {
+ cols = bms_del_member(cols, i + 1);
+ continue;
+ }
nliveatts++;
}
@@ -1131,8 +1164,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
}
}
+ if (HeapTupleIsValid(cftuple))
ReleaseSysCache(cftuple);
- }
}
if (first)
@@ -1560,18 +1593,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns,
- data->include_generated_columns);
+ new_slot, data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
default:
Assert(false);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 34ec40b07e..b9a64d9c95 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,22 +225,19 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
TupleTableSlot *newslot, bool binary,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -251,8 +248,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns,
- bool include_generated_columns);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 3bb2301b43..60ad5751bc 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1247,7 +1247,7 @@ $node_publisher->wait_for_catchup('sub1');
is( $node_subscriber->safe_psql(
'postgres', "SELECT * FROM test_mix_4 ORDER BY a"),
qq(1|2||
-3|4||),
+3|4||4),
'replication with multiple publications with the same column list');
# TEST: With a table included in multiple publications with different column
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-19 06:03 Shlok Kyal <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shlok Kyal @ 2024-07-19 06:03 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, 19 Jul 2024 at 04:59, Peter Smith <[email protected]> wrote:
>
> Hi, here are some review comments for patch v19-0003
>
> ======
> src/backend/catalog/pg_publication.c
>
> 1.
> /*
> * Translate a list of column names to an array of attribute numbers
> * and a Bitmapset with them; verify that each attribute is appropriate
> * to have in a publication column list (no system or generated attributes,
> * no duplicates). Additional checks with replica identity are done later;
> * see pub_collist_contains_invalid_column.
> *
> * Note that the attribute numbers are *not* offset by
> * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
> * is okay.
> */
> static void
> publication_translate_columns(Relation targetrel, List *columns,
> int *natts, AttrNumber **attrs)
>
> ~
>
> I though the above comment ought to change: /or generated
> attributes/or virtual generated attributes/
>
> IIRC this was already addressed back in v16, but somehow that fix has
> been lost (???).
Modified the comment
> ======
> src/backend/replication/logical/tablesync.c
>
> fetch_remote_table_info:
> nitpick - missing end space in this comment /* TODO: use
> ATTRIBUTE_GENERATED_VIRTUAL*/
>
Fixed
> ======
>
> 2.
> (in patch v19-0001)
> +# tab3:
> +# publisher-side tab3 has generated col 'b'.
> +# subscriber-side tab3 has generated col 'b', using a different computation.
>
> (here, in patch v19-0003)
> # tab3:
> -# publisher-side tab3 has generated col 'b'.
> -# subscriber-side tab3 has generated col 'b', using a different computation.
> +# publisher-side tab3 has stored generated col 'b' but
> +# subscriber-side tab3 has DIFFERENT COMPUTATION stored generated col 'b'.
>
> It has become difficult to review these TAP tests, particularly when
> different patches are modifying the same comment. e.g. I post
> suggestions to modify comments for patch 0001. Those get addressed OK,
> only to vanish in subsequent patches like has happened in the above
> example.
>
> Really this patch 0003 was only supposed to add the word "stored", not
> revert the entire comment to something from an earlier version. Please
> take care that all comment changes are carried forward correctly from
> one patch to the next.
Fixed
I have addressed the comment in v20-0003 patch. Please refer [1].
[1]: https://www.postgresql.org/message-id/CANhcyEUzUurrX38HGvG30gV92YDz6WmnnwNRYMVY4tiga-8KZg%40mail.gma...
Thanks and Regards,
Shlok Kyal
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-22 23:23 Peter Smith <[email protected]>
parent: Shlok Kyal <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-22 23:23 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Jul 19, 2024 at 4:01 PM Shlok Kyal <[email protected]> wrote:
>
> On Thu, 18 Jul 2024 at 13:55, Peter Smith <[email protected]> wrote:
> >
> > Hi, here are some review comments for v19-0002
> > ======
> > src/test/subscription/t/004_sync.pl
> >
> > 1.
> > This new test is not related to generated columns. IIRC, this is just
> > some test that we discovered missing during review of this thread. As
> > such, I think this change can be posted/patched separately from this
> > thread.
> >
> I have removed the test for this thread.
>
> I have also addressed the remaining comments for v19-0002 patch.
Hi, I have no more review comments for patch v20-0002 at this time.
I saw that the above test was removed from this thread as suggested,
but I could not find that any new thread was started to propose this
valuable missing test.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-26 12:23 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Shubham Khanna @ 2024-07-26 12:23 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Jul 18, 2024 at 10:47 AM Peter Smith <[email protected]> wrote:
>
> Hi Shubham, here are my review comments for patch v19-0001.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 1.
> /*
> * Columns included in the publication, or NULL if all columns are
> * included implicitly. Note that the attnums in this bitmap are not
> + * publication and include_generated_columns option: other reasons should
> + * be checked at user side. Note that the attnums in this bitmap are not
> * shifted by FirstLowInvalidHeapAttributeNumber.
> */
> Bitmapset *columns;
> You replied [1] "The attached Patches contain all the suggested
> changes." but as I previously commented [2, #1], since there is no
> change to the interpretation of the 'columns' BMS caused by this
> patch, then I expected this comment would be unchanged (i.e. same as
> HEAD code). But this fix was missed in v19-0001.
>
> OTOH, if you do think there was a reason to change the comment then
> the above is still not good because "are not publication and
> include_generated_columns option" wording doesn't make sense.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> Observation -- I added (in nitpicks diffs) some more comments for
> 'tab1' (to make all comments consistent with the new tests added). But
> when I was doing that I observed that tab1 and tab3 test scenarios are
> very similar. It seems only the subscription parameter is not
> specified (so 'include_generated_cols' default wll be tested). IIRC
> the default for that parameter is "false", so tab1 is not really
> testing that properly -- e.g. I thought maybe to test the default
> parameter it's better the subscriber-side 'b' should be not-generated?
> But doing that would make 'tab1' the same as 'tab2'. Anyway, something
> seems amiss -- it seems either something is not tested or is duplicate
> tested. Please revisit what the tab1 test intention was and make sure
> we are doing the right thing for it...
>
> ======
> 99.
> The attached nitpicks diff patch has some tweaked comments.
>
> ======
> [1] https://www.postgresql.org/message-id/CAHv8Rj%2BR0cj%3Dz1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A%40mail...
> [2] https://www.postgresql.org/message-id/CAHut%2BPtVfrbx0jb42LCmS%3D-LcMTtWxm%2BvhaoArkjg7Z0mvuXbg%40ma...
The attached Patches contain all the suggested changes.
v21-0001 - Addressed the comments.
v21-0002 - Added the TAP Tests for 011_generated.pl file and modified
the patch accordingly.
v21-0003 - Added the TAP Tests for 011_generated.pl file and modified
the patch accordingly.
v21-0004- Rebased the Patch.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v21-0001-Enable-support-for-include_generated_columns-opt.patch (97.7K, ../../CAHv8RjJ4oyaWRaTDHUSh=L3=tZGfAi+FAmQtqsurF3C_fCxYYg@mail.gmail.com/2-v21-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From d65517e1fa607978d9cf2b2951dc3a0452c2b922 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Wed, 24 Jul 2024 11:22:07 +0530
Subject: [PATCH v21 1/4] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION
pub1 WITH (include_generated_columns = true, copy_data = false)"
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 +++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 ++
contrib/test_decoding/test_decoding.c | 26 +-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 ++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++-
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 ++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++++-----
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 296 +++++++++++++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 673 insertions(+), 138 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 79cd599692..3320c25a60 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3322,6 +3322,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6540,8 +6551,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..819a124c63 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ec96b5fe85..7564173bee 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4376,6 +4376,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..e99f528e39 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5167,6 +5177,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..f7b8d59413 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 891face1b6..51fe260fdb 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3363,7 +3363,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..fe621074a7 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -14,10 +14,16 @@ my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
$node_publisher->start;
+# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
$node_subscriber->start;
+# All subscribers on this node will use parameter include_generated_columns = true
+my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$node_subscriber2->init;
+$node_subscriber2->start;
+
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
@@ -27,26 +33,148 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
+);
+
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', with different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)"
+);
+
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+# tab_order:
+# publisher-side has generated cols 'b' and 'c' but
+# subscriber-side has non-generated col 'b', and generated-col 'c'
+# where columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab_alter:
+# for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1, tab_gen_to_gen, tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen, tab_order");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (include_generated_columns = true, copy_data = false)"
+);
+
+#####################
# Wait for initial sync of all subscriptions
+#####################
+
$node_subscriber->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+# gen-to-gen
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+
+# gen-to-nogen
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'generated columns initial sync, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+
+# missing-to_gen
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is( $result, qq(), 'generated column initial sync');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(), 'unsubscribed table initial data');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -62,8 +190,174 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#####################
+# TEST tab_gen_to_gen
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', using a different computation.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# sub1: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+#$node_publisher->wait_for_catchup('sub1_gen_to_gen');
+$node_publisher->wait_for_catchup('sub1');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(1|21
+2|22
+3|23
+4|24
+5|25),
+ 'confirm generated columns are NOT replicated, when include_generated_columns=false'
+);
+
+# sub2: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+$node_publisher->wait_for_catchup('sub2_gen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
+#####################
+# TEST tab_gen_to_nogen
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# sub1: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('sub1');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# sub2: (include_generated_columns = true)
+# Confirm that col 'b' is replicated.
+$node_publisher->wait_for_catchup('sub2_gen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#####################
+# TEST tab_missing_to_gen
+#
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# sub1: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('sub1');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|2
+2|4
+3|6
+4|8
+5|10),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# sub2: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('sub2_gen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+#####################
+# TEST tab_order:
+#
+# publisher-side cols 'b' and 'c' are generated
+# subscriber-side col 'b' is not generated and col 'c' is generated.
+# But pub/sub table cols are in different order.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_order VALUES (4), (5)");
+
+# sub2: (include_generated_columns = true)
+# Confirm depsite different orders replication occurs to the correct columns
+$node_publisher->wait_for_catchup('sub2_gen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is( $result, qq(4|8|88
+5|10|110), 'replicate generated columns with different order on subscriber');
+
+#####################
+# TEST tab_alter
+#
+# Add new table to existing publication, then
+# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#####################
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab_alter");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2_gen_to_gen REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('sub2_gen_to_gen');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66), 'add new table to existing publication');
+
+#####################
+# TEST tabl_alter
+#
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#####################
+
+# change a gencol to a nogen col
+$node_subscriber2->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN c DROP EXPRESSION");
+
+# insert some data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (4), (5)");
+
+# confirmed replication now works for the subscriber nogen col
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
+#####################
# try it with a subscriber-side trigger
+
$node_subscriber->safe_psql(
'postgres', q{
CREATE FUNCTION tab1_trigger_func() RETURNS trigger
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
[application/octet-stream] v21-0002-Support-replication-of-generated-column-during-i.patch (25.9K, ../../CAHv8RjJ4oyaWRaTDHUSh=L3=tZGfAi+FAmQtqsurF3C_fCxYYg@mail.gmail.com/3-v21-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From ec26445db8f86bb950ddcb722706dfc33ebbf839 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Thu, 25 Jul 2024 16:44:18 +0530
Subject: [PATCH v21 2/4] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is
replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when
'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Summary:
when (include_generated_columns = true)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column: The
publisher generated column value is copied.
* publisher generated column => subscriber generated column: The
publisher generated column value is not copied. The subscriber
generated column will be filled with the subscriber-side computed or
default data.
when (include_generated_columns = false)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column:
Publisher generated column is not replicated. The subscriber column
will be filled with the subscriber-side default data.
* publisher generated column => subscriber generated column: Publisher
generated column is not replicated. The subscriber generated column
will be filed with the subscriber-side computed or default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 --
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/tablesync.c | 139 ++++++++++++++++----
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 2 -
src/test/regress/sql/subscription.sql | 1 -
src/test/subscription/t/011_generated.pl | 126 +++++++++++++++---
8 files changed, 229 insertions(+), 62 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 819a124c63..18b2a8e040 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -450,20 +450,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..9de0b75330 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e03e761392..2e90d42bdc 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -693,20 +694,67 @@ process_syncing_tables(XLogRecPtr current_lsn)
/*
* Create list of columns for COPY based on logical relation mapping.
+ * Exclude columns that are subscription table generated columns.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *localgenlist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
+
+ /*
+ * This loop checks for generated columns of the subscription table.
+ */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int remote_attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (remote_attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if (!remotegenlist[remote_attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'localgenlist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ localgenlist[remote_attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY, excluding columns that are
+ * subscription table generated columns.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if (!localgenlist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
+ pfree(localgenlist);
return attnamelist;
}
@@ -791,19 +839,21 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
+ int server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
lrel->nspname = nspname;
lrel->relname = relname;
@@ -851,7 +901,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
WalRcvExecResult *pubres;
TupleTableSlot *tslot;
@@ -948,18 +998,31 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey)");
+
+ if(server_version >= 120000)
+ appendStringInfo(&cmd, ", a.attgenerated != ''");
+
+ appendStringInfo(&cmd,
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if (server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
+
+ if (!gencols_allowed)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1036,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1069,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1081,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist_res = remotegenlist;
walrcv_clear_result(res);
/*
@@ -1037,7 +1103,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* 3) one of the subscribed publications is declared as TABLES IN SCHEMA
* that includes this relation
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
StringInfoData pub_names;
@@ -1123,10 +1189,13 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
+ bool gencol_copy_needed = false;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1204,31 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Check if the remote table has any generated columns that should be copied.
+ */
+ if (MySubscription->includegencols)
+ {
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if (remotegenlist[i])
+ {
+ gencol_copy_needed = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * Regular table with no row filter and copy of generated columns is
+ * not necessary.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL && !gencol_copy_needed)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1173,13 +1262,20 @@ copy_table(Relation rel)
* (SELECT ...), but we can't just do SELECT * because we need to not
* copy generated columns. For tables with any row filters, build a
* SELECT query with OR'ed row filters for COPY.
+ *
+ * We also need to use this same COPY (SELECT ...) syntax when
+ * 'include_generated_columns' is specified as true and the remote
+ * table has generated columns, because copy of generated columns is
+ * not supported by the normal COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_node(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1333,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 3e08be39b7..e6eba1bea0 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,8 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7f7057d1b4..c88e7966bf 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,7 +59,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index fe621074a7..8ff3f4ad05 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,6 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
@@ -65,6 +66,14 @@ $node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab_gen_to_missing (a int)"
);
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
# publisher-side col 'b' is missing.
# subscriber-side col 'b' is generated.
$node_publisher->safe_psql('postgres',
@@ -113,6 +122,8 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
@@ -121,14 +132,14 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR TABLE tab1, tab_gen_to_gen, tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen, tab_order");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1, tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen, tab_order");
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (include_generated_columns = true)"
);
#####################
@@ -149,7 +160,9 @@ is( $result, qq(1|21
2|22
3|23), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync, when include_generated_columns=true');
# gen-to-nogen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
@@ -157,7 +170,9 @@ is( $result, qq(1|
2|
3|), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync, when include_generated_columns=true');
# missing-to_gen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
@@ -165,11 +180,15 @@ is( $result, qq(1|2
2|4
3|6), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync, when include_generated_columns=true');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
-is( $result, qq(), 'generated column initial sync');
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'generated column initial sync');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
@@ -202,7 +221,6 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_gen VALUES (4), (
# sub1: (include_generated_columns = false)
# Confirm that col 'b' is not replicated.
-#$node_publisher->wait_for_catchup('sub1_gen_to_gen');
$node_publisher->wait_for_catchup('sub1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
@@ -221,7 +239,10 @@ is( $result, qq(1|21
$node_publisher->wait_for_catchup('sub2_gen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
-is( $result, qq(4|24
+is( $result, qq(1|21
+2|22
+3|23
+4|24
5|25),
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
@@ -254,11 +275,76 @@ is( $result, qq(1|
$node_publisher->wait_for_catchup('sub2_gen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm generated columns are replicated when the subscriber-side column is not generated'
);
+#####################
+# TEST tab_gen_to_missing
+#
+# publisher-side col 'b' is generated.
+# subscriber-side col 'b' is missing
+#####################
+
+# sub1: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub_gen_to_missing FOR TABLE tab_gen_to_missing");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION pub_gen_to_missing"
+);
+$node_publisher->wait_for_catchup('sub1_gen_to_missing');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'missing generated column, include_generated_columns = false');
+
+# sub2: (include_generated_columns = true)
+# Confirm that col 'b' s not replicated and it will throw an error.
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset2 = -s $node_subscriber2->logfile;
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION pub_gen_to_missing with (include_generated_columns = true)"
+);
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
+#####################
+# TEST tab_nogen_to_gen
+#
+# publisher-side col 'b' is not-generated.
+# subscriber-side col 'b' is generated
+#####################
+
+# sub1: (include_generated_columns = false)
+# Confirm that col 'b' s not replicated and it will throw an error.
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub_nogen_to_gen FOR TABLE tab_nogen_to_gen");
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION pub_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+
+# sub2: (include_generated_columns = true)
+# Confirm that col 'b' s not replicated and it will throw an error.
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION pub_nogen_to_gen WITH (include_generated_columns = true)"
+);
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset2);
+
#####################
# TEST tab_missing_to_gen
#
@@ -287,7 +373,10 @@ is( $result, qq(1|2
$node_publisher->wait_for_catchup('sub2_gen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm when publisher col is missing, subscriber generated columns are generated as normal'
);
@@ -309,7 +398,10 @@ $node_publisher->wait_for_catchup('sub2_gen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
-is( $result, qq(4|8|88
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
5|10|110), 'replicate generated columns with different order on subscriber');
#####################
@@ -326,9 +418,9 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->wait_for_catchup('sub2_gen_to_gen');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
-is( $result, qq(1||22
-2||44
-3||66), 'add new table to existing publication');
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'add new table to existing publication');
#####################
# TEST tabl_alter
@@ -348,9 +440,9 @@ $node_publisher->safe_psql('postgres',
# confirmed replication now works for the subscriber nogen col
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
-is( $result, qq(1||22
-2||44
-3||66
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
4|8|8
5|10|10), 'after drop generated column expression');
--
2.41.0.windows.3
[application/octet-stream] v21-0004-Improve-include-generated-column-option-handling.patch (15.3K, ../../CAHv8RjJ4oyaWRaTDHUSh=L3=tZGfAi+FAmQtqsurF3C_fCxYYg@mail.gmail.com/4-v21-0004-Improve-include-generated-column-option-handling.patch)
download | inline diff:
From 2860e4469aaf6d7cb2682dcc6a06854a3db1b859 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Fri, 26 Jul 2024 12:42:42 +0530
Subject: [PATCH v21 4/4] Improve include generated column option handling by
using bms
Improve include generated column option handling by using bms.
---
src/backend/replication/logical/proto.c | 72 +++-------------
src/backend/replication/pgoutput/pgoutput.c | 95 ++++++++++++++-------
src/include/replication/logicalproto.h | 12 +--
3 files changed, 80 insertions(+), 99 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index cad1b76e7a..6b085e555c 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,12 +30,10 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -414,8 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -427,8 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -461,8 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -483,13 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -539,7 +532,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -559,8 +552,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
/*
@@ -676,7 +668,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
char *relname;
@@ -698,7 +690,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns, include_generated_columns);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -775,8 +767,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -793,15 +784,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -823,15 +805,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -950,8 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
- bool include_generated_columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -969,15 +941,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -999,15 +962,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index c02de23743..0c4c7ac5ba 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,8 +86,7 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -165,8 +164,12 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns included in the publication, or NULL if all columns are
- * included implicitly. Note that the attnums in this bitmap are not
+ * Columns should be publicated, or NULL if all columns are included
+ * implicitly. This bitmap only considers the column list of the
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
@@ -744,13 +747,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -764,7 +765,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -784,15 +785,6 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -806,7 +798,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -1032,6 +1024,36 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Prepare new column list bitmap. This includes all the columns of the table.
+ */
+static Bitmapset *
+prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
+ TupleDesc desc)
+{
+ Bitmapset *cols = NULL;
+ MemoryContext oldcxt = NULL;
+
+ pgoutput_ensure_entry_cxt(data, entry);
+ oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped */
+ if (att->attisdropped)
+ continue;
+
+ /* Iterate the cols until generated columns are found. */
+ cols = bms_add_member(cols, i + 1);
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+
+ return cols;
+}
+
/*
* Initialize the column list.
*/
@@ -1070,7 +1092,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
* if there are no column lists (even if other publications have a
* list).
*/
- if (!pub->alltables)
+ if (!pub->alltables || !data->include_generated_columns)
{
bool pub_no_list = true;
@@ -1091,9 +1113,10 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
+ }
/* Build the column list bitmap in the per-entry context. */
- if (!pub_no_list) /* when not null */
+ if (!pub_no_list || !data->include_generated_columns) /* when not null */
{
int i;
int nliveatts = 0;
@@ -1101,19 +1124,30 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
pgoutput_ensure_entry_cxt(data, entry);
- cols = pub_collist_to_bitmapset(cols, cfdatum,
+ if (!pub_no_list)
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
entry->entry_cxt);
+ else
+ cols = prepare_all_columns_bms(data, entry, desc);
/* Get the number of live attributes. */
for (i = 0; i < desc->natts; i++)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
+ /* Skip if the attribute is dropped */
if (att->attisdropped)
continue;
-
- if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
+ /*
+ * If column list contain generated column it will not replicate
+ * the table to the subscriber port.
+ */
+ if (att->attgenerated &&
+ (att->attgenerated != ATTRIBUTE_GENERATED_STORED ||
+ !data->include_generated_columns))
+ {
+ cols = bms_del_member(cols, i + 1);
+ }
nliveatts++;
}
@@ -1129,8 +1163,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
}
}
+ if (HeapTupleIsValid(cftuple))
ReleaseSysCache(cftuple);
- }
}
if (first)
@@ -1558,18 +1592,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns,
- data->include_generated_columns);
+ new_slot, data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
default:
Assert(false);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 34ec40b07e..b9a64d9c95 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,22 +225,19 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
TupleTableSlot *newslot, bool binary,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -251,8 +248,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns,
- bool include_generated_columns);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
--
2.34.1
[application/octet-stream] v21-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (14.9K, ../../CAHv8RjJ4oyaWRaTDHUSh=L3=tZGfAi+FAmQtqsurF3C_fCxYYg@mail.gmail.com/5-v21-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From 34e14aa0b0be5ee8b98870817b13f320ee00d4b0 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Wed, 24 Jul 2024 12:16:45 +0530
Subject: [PATCH v21 3/4] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
.../expected/generated_columns.out | 1 +
.../test_decoding/sql/generated_columns.sql | 4 +-
contrib/test_decoding/test_decoding.c | 15 ++++++-
doc/src/sgml/protocol.sgml | 7 ++--
doc/src/sgml/ref/create_subscription.sgml | 4 +-
src/backend/catalog/pg_publication.c | 18 +++++++--
src/backend/replication/logical/proto.c | 40 +++++++++++++++----
src/backend/replication/logical/tablesync.c | 8 +++-
src/backend/replication/pgoutput/pgoutput.c | 13 +++++-
src/test/subscription/t/011_generated.pl | 28 ++++++-------
10 files changed, 102 insertions(+), 36 deletions(-)
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index f3b26aa9e1..a79510705c 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -50,3 +50,4 @@ SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
stop
(1 row)
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 6d6d1d6564..997cdebc7e 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -19,4 +19,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..a847050f6e 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,8 +557,19 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
- continue;
+ if (attr->attgenerated)
+ {
+ /*
+ * Don't print generated columns when
+ * 'include_generated_columns' is false.
+ */
+ if (!include_generated_columns)
+ continue;
+
+ /* Don't print generated columns unless they are STORED. */
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
/*
* Don't print system columns, oid will already have been printed if
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 3320c25a60..a2713a95b1 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3326,9 +3326,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..71466b1583 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -490,9 +490,9 @@ compare_int16(const void *a, const void *b)
/*
* Translate a list of column names to an array of attribute numbers
* and a Bitmapset with them; verify that each attribute is appropriate
- * to have in a publication column list (no system or generated attributes,
- * no duplicates). Additional checks with replica identity are done later;
- * see pub_collist_contains_invalid_column.
+ * to have in a publication column list (no system or virtual generated
+ * attributes, no duplicates). Additional checks with replica identity
+ * are done later; see pub_collist_contains_invalid_column.
*
* Note that the attribute numbers are *not* offset by
* FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -520,6 +521,7 @@ publication_translate_columns(Relation targetrel, List *columns,
{
char *colname = strVal(lfirst(lc));
AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+ Form_pg_attribute att;
if (attnum == InvalidAttrNumber)
ereport(ERROR,
@@ -533,6 +535,13 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ att = TupleDescAttr(tupdesc, attnum - 1);
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("generated column \"%s\" is not STORED so cannot be used in a publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1228,6 +1237,9 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
attnums[nattnums++] = att->attnum;
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e694baca0a..cad1b76e7a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,8 +793,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -817,8 +823,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -957,8 +969,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -981,8 +999,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 2e90d42bdc..14a0aae416 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1014,7 +1014,13 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
{
bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
- if (!gencols_allowed)
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ /* TODO: use ATTRIBUTE_GENERATED_VIRTUAL */
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
appendStringInfo(&cmd, " AND a.attgenerated = ''");
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 4624649cd7..c02de23743 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -784,8 +784,14 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -1106,6 +1112,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8ff3f4ad05..b61bbdd137 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -38,8 +38,8 @@ $node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side has generated col 'b'.
-# subscriber-side has generated col 'b', with different computation.
+# publisher-side has stored generated col 'b'.
+# subscriber-side has stored generated col 'b', with different computation.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
@@ -47,14 +47,14 @@ $node_subscriber->safe_psql('postgres',
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
$node_subscriber2->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side col 'b' is missing.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -67,7 +67,7 @@ $node_subscriber2->safe_psql('postgres',
);
# publisher-side has non-generated col 'b'.
-# subscriber-side has generated col 'b'.
+# subscriber-side has stored generated col 'b'.
$node_publisher->safe_psql('postgres', "CREATE TABLE tab_nogen_to_gen (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
@@ -87,8 +87,8 @@ $node_subscriber2->safe_psql('postgres',
);
# tab_order:
-# publisher-side has generated cols 'b' and 'c' but
-# subscriber-side has non-generated col 'b', and generated-col 'c'
+# publisher-side has stored generated cols 'b' and 'c' but
+# subscriber-side has non-generated col 'b', and stored generated-col 'c'
# where columns on publisher/subscriber are in a different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -212,7 +212,7 @@ is( $result, qq(1|22|
#####################
# TEST tab_gen_to_gen
#
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side has generated col 'b', using a different computation.
#####################
@@ -250,7 +250,7 @@ is( $result, qq(1|21
#####################
# TEST tab_gen_to_nogen
#
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side has non-generated col 'b'.
#####################
@@ -286,7 +286,7 @@ is( $result, qq(1|2
#####################
# TEST tab_gen_to_missing
#
-# publisher-side col 'b' is generated.
+# publisher-side col 'b' is stored generated.
# subscriber-side col 'b' is missing
#####################
@@ -384,8 +384,8 @@ is( $result, qq(1|2
#####################
# TEST tab_order:
#
-# publisher-side cols 'b' and 'c' are generated
-# subscriber-side col 'b' is not generated and col 'c' is generated.
+# publisher-side cols 'b' and 'c' are stored generated
+# subscriber-side col 'b' is not generated and col 'c' is stored generated.
# But pub/sub table cols are in different order.
#####################
@@ -425,8 +425,8 @@ is( $result, qq(1|2|22
#####################
# TEST tabl_alter
#
-# Drop the generated column's expression on subscriber side.
-# This changes the generated column into a non-generated column.
+# Drop the stored generated column's expression on subscriber side.
+# This changes the stored generated column into a non-generated column.
#####################
# change a gencol to a nogen col
--
2.41.0.windows.3
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-29 07:26 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-07-29 07:26 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Thanks for the patch updates.
Here are my review comments for v21-0001.
I think this patch is mostly OK now except there are still some
comments about the TAP test.
======
Commit Message
0.
Using Create Subscription:
CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION
pub1 WITH (include_generated_columns = true, copy_data = false)"
If you are going to give an example, I think a gen-to-nogen example
would be a better choice. That's because the gen-to-gen (as you have
here) is not going to replicate anything due to the subscriber-side
column being generated.
======
src/test/subscription/t/011_generated.pl
1.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
* 22) STORED, c int)"
+);
The subscriber2 node was intended only for all the tables where we
need include_generated_columns to be true. Mostly that is the
combination tests. (tab_gen_to_nogen, tab_nogen_to_gen, etc) OTOH,
table 'tab1' already existed. I don't think we need to bother
subscribing to tab1 from subscriber2 because every combination is
already covered by the combination tests. Let's leave this one alone.
~~~
2.
Huh? Where is the "tab_nogen_to_gen" combination test that I sent to
you off-list?
~~~
3.
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED,
a int, b int)"
+);
Maybe you can test 'tab_order' on both subscription nodes but I think
it is overkill. IMO it is enough to test it on subscription2.
~~~
4.
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a
* 22) STORED)"
+);
Ditto above. Maybe you can test 'tab_order' on both subscription nodes
but I think it is overkill. IMO it is enough to test it on
subscription2.
~~~
5.
Don't forget to add initial data for the missing nogen_to_gen table/test.
~~~
6.
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1, tab_gen_to_gen,
tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen, tab_order");
+
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
);
It is not a bad idea to reduce the number of publications as you have
done, but IMO jamming all the tables into 1 publication is too much
because it makes it less understandable instead of simpler.
How about this:
- leave the 'pub1' just for 'tab1'.
- have a 'pub_combo' for publication all the gen_to_nogen,
nogen_to_gen etc combination tests.
- and a 'pub_misc' for any other misc tables like tab_order.
~~~
7.
+#####################
# Wait for initial sync of all subscriptions
+#####################
I think you should write a note here that you have deliberately set
copy_data = false because COPY and include_generated_columns are not
allowed at the same time for patch 0001. And that is why all expected
results on subscriber2 will be empty. Also, say this limitation will
be changed in patch 0002.
~~~
(I didn't yet check 011_generated.pl file results beyond this point...
I'll wait for v22-0001 to review further)
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-07-31 08:42 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Shubham Khanna @ 2024-07-31 08:42 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Jul 29, 2024 at 12:57 PM Peter Smith <[email protected]> wrote:
>
> Thanks for the patch updates.
>
> Here are my review comments for v21-0001.
>
> I think this patch is mostly OK now except there are still some
> comments about the TAP test.
>
> ======
> Commit Message
>
> 0.
> Using Create Subscription:
> CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION
> pub1 WITH (include_generated_columns = true, copy_data = false)"
>
> If you are going to give an example, I think a gen-to-nogen example
> would be a better choice. That's because the gen-to-gen (as you have
> here) is not going to replicate anything due to the subscriber-side
> column being generated.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> 1.
> +$node_subscriber2->safe_psql('postgres',
> + "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
> * 22) STORED, c int)"
> +);
>
> The subscriber2 node was intended only for all the tables where we
> need include_generated_columns to be true. Mostly that is the
> combination tests. (tab_gen_to_nogen, tab_nogen_to_gen, etc) OTOH,
> table 'tab1' already existed. I don't think we need to bother
> subscribing to tab1 from subscriber2 because every combination is
> already covered by the combination tests. Let's leave this one alone.
>
>
> ~~~
>
> 2.
> Huh? Where is the "tab_nogen_to_gen" combination test that I sent to
> you off-list?
>
> ~~~
>
> 3.
> +$node_subscriber->safe_psql('postgres',
> + "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED,
> a int, b int)"
> +);
>
> Maybe you can test 'tab_order' on both subscription nodes but I think
> it is overkill. IMO it is enough to test it on subscription2.
>
> ~~~
>
> 4.
> +$node_subscriber->safe_psql('postgres',
> + "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a
> * 22) STORED)"
> +);
>
> Ditto above. Maybe you can test 'tab_order' on both subscription nodes
> but I think it is overkill. IMO it is enough to test it on
> subscription2.
>
> ~~~
>
> 5.
> Don't forget to add initial data for the missing nogen_to_gen table/test.
>
> ~~~
>
> 6.
> $node_publisher->safe_psql('postgres',
> - "CREATE PUBLICATION pub1 FOR ALL TABLES");
> + "CREATE PUBLICATION pub1 FOR TABLE tab1, tab_gen_to_gen,
> tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen, tab_order");
> +
> $node_subscriber->safe_psql('postgres',
> "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
> );
>
> It is not a bad idea to reduce the number of publications as you have
> done, but IMO jamming all the tables into 1 publication is too much
> because it makes it less understandable instead of simpler.
>
> How about this:
> - leave the 'pub1' just for 'tab1'.
> - have a 'pub_combo' for publication all the gen_to_nogen,
> nogen_to_gen etc combination tests.
> - and a 'pub_misc' for any other misc tables like tab_order.
>
> ~~~
>
> 7.
> +#####################
> # Wait for initial sync of all subscriptions
> +#####################
>
> I think you should write a note here that you have deliberately set
> copy_data = false because COPY and include_generated_columns are not
> allowed at the same time for patch 0001. And that is why all expected
> results on subscriber2 will be empty. Also, say this limitation will
> be changed in patch 0002.
>
> ~~~
>
> (I didn't yet check 011_generated.pl file results beyond this point...
> I'll wait for v22-0001 to review further)
The attached Patches contain all the suggested changes.
v22-0001 - Addressed the comments.
v22-0002 - Rebased the Patch.
v22-0003 - Rebased the Patch.
v22-0004 - Rebased the Patch.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v22-0003-Fix-behaviour-for-Virtual-Generated-columns.patch (15.0K, ../../CAHv8RjJsGWETA9U53iRiV2+VGtnHamEJ5PKMHUcfat269kQaSQ@mail.gmail.com/2-v22-0003-Fix-behaviour-for-Virtual-Generated-columns.patch)
download | inline diff:
From 4fd3c5682c83db16e193dc904b2f3f2f70ec2383 Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Wed, 31 Jul 2024 12:03:13 +0530
Subject: [PATCH v22 3/4] Fix behaviour for Virtual Generated columns
Currently during tablesync Virtual generated columns are also
replicated. Also during decoding a 'null' value appears for virtual
generated column. We are not supporting replication of virtual generated
columns for now. This patch fixes the behaviour for the same.
This patch has a dependency on Virtual Generated Columns
https://www.postgresql.org/message-id/flat/787a962749e7a822a44803ffbbdf021d8573ff53.camel%40post.pl#b64569231c9e1768e07f6bdc36c4070b
---
.../expected/generated_columns.out | 1 +
.../test_decoding/sql/generated_columns.sql | 4 +-
contrib/test_decoding/test_decoding.c | 15 ++++++-
doc/src/sgml/protocol.sgml | 7 ++--
doc/src/sgml/ref/create_subscription.sgml | 4 +-
src/backend/catalog/pg_publication.c | 18 +++++++--
src/backend/replication/logical/proto.c | 40 +++++++++++++++----
src/backend/replication/logical/tablesync.c | 8 +++-
src/backend/replication/pgoutput/pgoutput.c | 13 +++++-
src/test/subscription/t/011_generated.pl | 28 ++++++-------
10 files changed, 102 insertions(+), 36 deletions(-)
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index f3b26aa9e1..a79510705c 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -50,3 +50,4 @@ SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
stop
(1 row)
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index 6d6d1d6564..997cdebc7e 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -19,4 +19,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
DROP TABLE gencoltable;
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+
+-- TODO: Add tests related to decoding of VIRTUAL GENERATED columns
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index eaa3dbf9db..a847050f6e 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -557,8 +557,19 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
if (attr->attisdropped)
continue;
- if (attr->attgenerated && !include_generated_columns)
- continue;
+ if (attr->attgenerated)
+ {
+ /*
+ * Don't print generated columns when
+ * 'include_generated_columns' is false.
+ */
+ if (!include_generated_columns)
+ continue;
+
+ /* Don't print generated columns unless they are STORED. */
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
/*
* Don't print system columns, oid will already have been printed if
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 3320c25a60..a2713a95b1 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3326,9 +3326,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<term>include_generated_columns</term>
<listitem>
<para>
- Boolean option to enable generated columns. This option controls
- whether generated columns should be included in the string
- representation of tuples during logical decoding in PostgreSQL.
+ Boolean option to enable <literal>STORED</literal> generated columns.
+ This option controls whether <literal>STORED</literal> generated columns
+ should be included in the string representation of tuples during logical
+ decoding in PostgreSQL.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8fb4491b65..91e33174dc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -433,8 +433,8 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
<term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
<listitem>
<para>
- Specifies whether the generated columns present in the tables
- associated with the subscription should be replicated.
+ Specifies whether the <literal>STORED</literal> generated columns present
+ in the tables associated with the subscription should be replicated.
The default is <literal>false</literal>.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f611148472..71466b1583 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -490,9 +490,9 @@ compare_int16(const void *a, const void *b)
/*
* Translate a list of column names to an array of attribute numbers
* and a Bitmapset with them; verify that each attribute is appropriate
- * to have in a publication column list (no system or generated attributes,
- * no duplicates). Additional checks with replica identity are done later;
- * see pub_collist_contains_invalid_column.
+ * to have in a publication column list (no system or virtual generated
+ * attributes, no duplicates). Additional checks with replica identity
+ * are done later; see pub_collist_contains_invalid_column.
*
* Note that the attribute numbers are *not* offset by
* FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
@@ -506,6 +506,7 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -520,6 +521,7 @@ publication_translate_columns(Relation targetrel, List *columns,
{
char *colname = strVal(lfirst(lc));
AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+ Form_pg_attribute att;
if (attnum == InvalidAttrNumber)
ereport(ERROR,
@@ -533,6 +535,13 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
+ att = TupleDescAttr(tupdesc, attnum - 1);
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("generated column \"%s\" is not STORED so cannot be used in a publication column list",
+ colname));
+
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1228,6 +1237,9 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
attnums[nattnums++] = att->attnum;
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e694baca0a..cad1b76e7a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -793,8 +793,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -817,8 +823,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -957,8 +969,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -981,8 +999,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 2e90d42bdc..14a0aae416 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1014,7 +1014,13 @@ fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
{
bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
- if (!gencols_allowed)
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ /* TODO: use ATTRIBUTE_GENERATED_VIRTUAL */
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }
+ else
appendStringInfo(&cmd, " AND a.attgenerated = ''");
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 4624649cd7..c02de23743 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -784,8 +784,14 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated && !include_generated_columns)
- continue;
+ if (att->attgenerated)
+ {
+ if (!include_generated_columns)
+ continue;
+
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ }
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -1106,6 +1112,9 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
if (att->attisdropped)
continue;
+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+
nliveatts++;
}
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index e128567fe1..171e555854 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -34,8 +34,8 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
-# publisher-side has generated col 'b'.
-# subscriber-side has generated col 'b', with different computation.
+# publisher-side has stored generated col 'b'.
+# subscriber-side has stored generated col 'b', with different computation.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
$node_subscriber->safe_psql('postgres',
@@ -43,14 +43,14 @@ $node_subscriber->safe_psql('postgres',
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
$node_subscriber2->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side col 'b' is missing.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -64,7 +64,7 @@ $node_subscriber2->safe_psql('postgres',
);
# publisher-side has non-generated col 'b'.
-# subscriber-side has generated col 'b'.
+# subscriber-side has stored generated col 'b'.
$node_publisher->safe_psql('postgres', "CREATE TABLE tab_nogen_to_gen (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
@@ -85,8 +85,8 @@ $node_subscriber2->safe_psql('postgres',
# tab_order:
-# publisher-side has generated cols 'b' and 'c' but
-# subscriber-side has non-generated col 'b', and generated-col 'c'
+# publisher-side has stored generated cols 'b' and 'c' but
+# subscriber-side has non-generated col 'b', and stored generated-col 'c'
# where columns on publisher/subscriber are in a different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
@@ -216,7 +216,7 @@ is( $result, qq(1|22|
#####################
# TEST tab_gen_to_gen
#
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side has generated col 'b', using a different computation.
#####################
@@ -255,7 +255,7 @@ is( $result, qq(1|21
#####################
# TEST tab_gen_to_nogen
#
-# publisher-side has generated col 'b'.
+# publisher-side has stored generated col 'b'.
# subscriber-side has non-generated col 'b'.
#####################
@@ -332,7 +332,7 @@ $node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo2");
#####################
# TEST tab_gen_to_missing
#
-# publisher-side col 'b' is generated.
+# publisher-side col 'b' is stored generated.
# subscriber-side col 'b' is missing
#####################
@@ -404,8 +404,8 @@ $node_subscriber2->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub2_nogen_t
#####################
# TEST tab_order:
#
-# publisher-side cols 'b' and 'c' are generated
-# subscriber-side col 'b' is not generated and col 'c' is generated.
+# publisher-side cols 'b' and 'c' are stored generated
+# subscriber-side col 'b' is not generated and col 'c' is stored generated.
# But pub/sub table cols are in different order.
#####################
@@ -445,8 +445,8 @@ is( $result, qq(1|2|22
#####################
# TEST tabl_alter
#
-# Drop the generated column's expression on subscriber side.
-# This changes the generated column into a non-generated column.
+# Drop the stored generated column's expression on subscriber side.
+# This changes the stored generated column into a non-generated column.
#####################
# change a gencol to a nogen col
--
2.34.1
[application/octet-stream] v22-0002-Support-replication-of-generated-column-during-i.patch (28.1K, ../../CAHv8RjJsGWETA9U53iRiV2+VGtnHamEJ5PKMHUcfat269kQaSQ@mail.gmail.com/3-v22-0002-Support-replication-of-generated-column-during-i.patch)
download | inline diff:
From 3cafc60b9073781d471d1b715b8350684b3d5d21 Mon Sep 17 00:00:00 2001
From: Shubham Khanna <[email protected]>
Date: Wed, 31 Jul 2024 11:47:08 +0530
Subject: [PATCH v22 2/4] Support replication of generated column during
initial sync
When 'copy_data' is true, during the initial sync, the data is
replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when
'include_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.
Summary:
when (include_generated_columns = true)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column: The
publisher generated column value is copied.
* publisher generated column => subscriber generated column: The
publisher generated column value is not copied. The subscriber
generated column will be filled with the subscriber-side computed or
default data.
when (include_generated_columns = false)
* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).
* publisher not-generated column => subscriber generated column: This
will give ERROR.
* publisher generated column => subscriber not-generated column:
Publisher generated column is not replicated. The subscriber column
will be filled with the subscriber-side default data.
* publisher generated column => subscriber generated column: Publisher
generated column is not replicated. The subscriber generated column
will be filed with the subscriber-side computed or default data.
---
doc/src/sgml/ref/create_subscription.sgml | 4 -
src/backend/commands/subscriptioncmds.c | 14 --
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/tablesync.c | 139 +++++++++++++---
src/include/replication/logicalrelation.h | 3 +-
src/test/regress/expected/subscription.out | 2 -
src/test/regress/sql/subscription.sql | 1 -
src/test/subscription/t/011_generated.pl | 171 +++++++++++++-------
8 files changed, 232 insertions(+), 104 deletions(-)
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index ee27a5873a..8fb4491b65 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -442,10 +442,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
has no effect; the subscriber column will be filled as normal with the
subscriber-side computed or default data.
</para>
- <para>
- This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
- set to <literal>false</literal>.
- </para>
</listitem>
</varlistentry>
</variablelist></para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 819a124c63..18b2a8e040 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -450,20 +450,6 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
-
- /*
- * Do additional checking for disallowed combination when copy_data and
- * include_generated_columns are true. COPY of generated columns is not
- * supported yet.
- */
- if (opts->copy_data && opts->include_generated_columns)
- {
- ereport(ERROR,
- errcode(ERRCODE_SYNTAX_ERROR),
- /*- translator: both %s are strings of the form "option = value" */
- errmsg("%s and %s are mutually exclusive options",
- "copy_data = true", "include_generated_columns = true"));
- }
}
/*
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 5de1531567..9de0b75330 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -205,7 +205,7 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
*
* Returns -1 if not found.
*/
-static int
+int
logicalrep_rel_att_by_name(LogicalRepRelation *remoterel, const char *attname)
{
int i;
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e03e761392..2e90d42bdc 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -693,20 +694,67 @@ process_syncing_tables(XLogRecPtr current_lsn)
/*
* Create list of columns for COPY based on logical relation mapping.
+ * Exclude columns that are subscription table generated columns.
*/
static List *
-make_copy_attnamelist(LogicalRepRelMapEntry *rel)
+make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
{
List *attnamelist = NIL;
- int i;
+ bool *localgenlist;
+ TupleDesc desc;
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
+
+ /*
+ * This loop checks for generated columns of the subscription table.
+ */
+ for (int i = 0; i < desc->natts; i++)
{
- attnamelist = lappend(attnamelist,
- makeString(rel->remoterel.attnames[i]));
+ int remote_attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ if (remote_attnum >= 0)
+ {
+ /*
+ * Check if the subscription table generated column has same
+ * name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if (!remotegenlist[remote_attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ /*
+ * 'localgenlist' records that this is a generated column in
+ * the subscription table. Later, we use this information to
+ * skip adding this column to the column list for COPY.
+ */
+ localgenlist[remote_attnum] = true;
+ }
}
+ /*
+ * Construct column list for COPY, excluding columns that are
+ * subscription table generated columns.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if (!localgenlist[i])
+ attnamelist = lappend(attnamelist,
+ makeString(rel->remoterel.attnames[i]));
+ }
+ pfree(localgenlist);
return attnamelist;
}
@@ -791,19 +839,21 @@ copy_read_data(void *outbuf, int minread, int maxread)
* qualifications to be used in the COPY command.
*/
static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist_res,
LogicalRepRelation *lrel, List **qual)
{
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
+ bool *remotegenlist;
int natt;
ListCell *lc;
Bitmapset *included_cols = NULL;
+ int server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
lrel->nspname = nspname;
lrel->relname = relname;
@@ -851,7 +901,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* We need to do this before fetching info about column names and types,
* so that we can skip columns that should not be replicated.
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
WalRcvExecResult *pubres;
TupleTableSlot *tslot;
@@ -948,18 +998,31 @@ fetch_remote_table_info(char *nspname, char *relname,
"SELECT a.attnum,"
" a.attname,"
" a.atttypid,"
- " a.attnum = ANY(i.indkey)"
+ " a.attnum = ANY(i.indkey)");
+
+ if(server_version >= 120000)
+ appendStringInfo(&cmd, ", a.attgenerated != ''");
+
+ appendStringInfo(&cmd,
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ if (server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 180000 && MySubscription->includegencols;
+
+ if (!gencols_allowed)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
+
res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
lengthof(attrRow), attrRow);
@@ -973,6 +1036,7 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ remotegenlist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
/*
* Store the columns as a list of names. Ignore those that are not
@@ -1005,6 +1069,8 @@ fetch_remote_table_info(char *nspname, char *relname,
if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
/* Should never happen. */
if (++natt >= MaxTupleAttributeNumber)
elog(ERROR, "too many columns in remote table \"%s.%s\"",
@@ -1015,7 +1081,7 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
lrel->natts = natt;
-
+ *remotegenlist_res = remotegenlist;
walrcv_clear_result(res);
/*
@@ -1037,7 +1103,7 @@ fetch_remote_table_info(char *nspname, char *relname,
* 3) one of the subscribed publications is declared as TABLES IN SCHEMA
* that includes this relation
*/
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ if (server_version >= 150000)
{
StringInfoData pub_names;
@@ -1123,10 +1189,13 @@ copy_table(Relation rel)
List *attnamelist;
ParseState *pstate;
List *options = NIL;
+ bool *remotegenlist;
+ bool gencol_copy_needed = false;
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &remotegenlist,
+ &lrel, &qual);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1135,11 +1204,31 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
/* Start copy on the publisher. */
initStringInfo(&cmd);
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Check if the remote table has any generated columns that should be copied.
+ */
+ if (MySubscription->includegencols)
+ {
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if (remotegenlist[i])
+ {
+ gencol_copy_needed = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * Regular table with no row filter and copy of generated columns is
+ * not necessary.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL && !gencol_copy_needed)
{
appendStringInfo(&cmd, "COPY %s",
quote_qualified_identifier(lrel.nspname, lrel.relname));
@@ -1173,13 +1262,20 @@ copy_table(Relation rel)
* (SELECT ...), but we can't just do SELECT * because we need to not
* copy generated columns. For tables with any row filters, build a
* SELECT query with OR'ed row filters for COPY.
+ *
+ * We also need to use this same COPY (SELECT ...) syntax when
+ * 'include_generated_columns' is specified as true and the remote
+ * table has generated columns, because copy of generated columns is
+ * not supported by the normal COPY.
*/
+ int i = 0;
+
appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_node(String, att_name, attnamelist)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i++)
appendStringInfoString(&cmd, ", ");
+ appendStringInfoString(&cmd, quote_identifier(strVal(att_name)));
}
appendStringInfoString(&cmd, " FROM ");
@@ -1237,7 +1333,6 @@ copy_table(Relation rel)
(void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
/* Do the copy */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index e687b40a56..797e66dfdb 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -41,7 +41,8 @@ typedef struct LogicalRepRelMapEntry
extern void logicalrep_relmap_update(LogicalRepRelation *remoterel);
extern void logicalrep_partmap_reset_relmap(LogicalRepRelation *remoterel);
-
+extern int logicalrep_rel_att_by_name(LogicalRepRelation *remoterel,
+ const char *attname);
extern LogicalRepRelMapEntry *logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode);
extern LogicalRepRelMapEntry *logicalrep_partition_open(LogicalRepRelMapEntry *root,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 3e08be39b7..e6eba1bea0 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,8 +99,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
ERROR: include_generated_columns requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 7f7057d1b4..c88e7966bf 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,7 +59,6 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
-CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
-- fail - include_generated_columns must be boolean
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 05b83f6bec..e128567fe1 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -58,12 +58,13 @@ $node_publisher->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab_gen_to_missing (a int)"
);
+
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab_gen_to_missing (a int)"
);
-# publisher-side col 'b' is missing.
-# subscriber-side col 'b' is generated.
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
$node_publisher->safe_psql('postgres', "CREATE TABLE tab_nogen_to_gen (a int, b int)");
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
@@ -125,10 +126,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION regress_pub1 FOR TABLE tab1");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen");
-$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub_nogen_to_gen FOR TABLE tab_nogen_to_gen");
-
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
@@ -139,26 +137,17 @@ $node_subscriber->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
);
-
$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true)"
);
$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
-);
-
-$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION regress_sub_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true)"
);
#####################
# Wait for initial sync of all subscriptions
#####################
-# Here, copy_data = false because COPY and include_generated_columns are not
-# allowed at the same time for patch 0001.
-# And that is why all expected results on subscriber2 will be empty.
-# This limitation will be changed in patch 0002.
$node_subscriber->wait_for_subscription_sync;
$node_subscriber2->wait_for_subscription_sync;
@@ -174,7 +163,9 @@ is( $result, qq(1|21
2|22
3|23), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync, when include_generated_columns=true');
# gen-to-nogen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
@@ -182,11 +173,9 @@ is( $result, qq(1|
2|
3|), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
-
-# nogen-to-gen
-$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync, when include_generated_columns=true');
# missing-to_gen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
@@ -194,11 +183,15 @@ is( $result, qq(1|2
2|4
3|6), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync, when include_generated_columns=true');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
-is( $result, qq(), 'generated column initial sync');
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'generated column initial sync');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
@@ -251,7 +244,10 @@ is( $result, qq(1|21
$node_publisher->wait_for_catchup('regress_sub_combo2');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
-is( $result, qq(4|24
+is( $result, qq(1|21
+2|22
+3|23
+4|24
5|25),
'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
);
@@ -284,35 +280,14 @@ is( $result, qq(1|
$node_publisher->wait_for_catchup('regress_sub_combo2');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm generated columns are replicated when the subscriber-side column is not generated'
);
-#####################
-# TEST tab_nogen_to_gen
-#
-# publisher-side has generated col 'b'.
-# subscriber-side has non-generated col 'b'.
-#####################
-
-# insert data
-$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
-
-# regress_sub_nogen_to_gen: (include_generated_columns = false)
-# Confirm that col 'b' is not replicated.
-$node_publisher->wait_for_catchup('regress_sub_nogen_to_gen');
-$result =
- $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
-is( $result, qq(4|88
-5|110),
- 'confirm generated columns are replicated when the subscriber-side column is not generated'
-);
-
-#Cleanup
-$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_nogen_to_gen");
-$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_nogen_to_gen");
-
#####################
# TEST tab_missing_to_gen
#
@@ -341,7 +316,10 @@ is( $result, qq(1|2
$node_publisher->wait_for_catchup('regress_sub_combo2');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
-is( $result, qq(4|8
+is( $result, qq(1|2
+2|4
+3|6
+4|8
5|10),
'confirm when publisher col is missing, subscriber generated columns are generated as normal'
);
@@ -351,6 +329,78 @@ $node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_combo");
$node_subscriber->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo");
$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo2");
+#####################
+# TEST tab_gen_to_missing
+#
+# publisher-side col 'b' is generated.
+# subscriber-side col 'b' is missing
+#####################
+
+# regress_sub1_gen_to_missing: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_gen_to_missing FOR TABLE tab_gen_to_missing");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_gen_to_missing"
+);
+$node_publisher->wait_for_catchup('regress_sub1_gen_to_missing');
+$result = $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'missing generated column, include_generated_columns = false');
+
+# regress_sub2_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' s not replicated and it will throw an error.
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset2 = -s $node_subscriber2->logfile;
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_gen_to_missing with (include_generated_columns = true)"
+);
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
+#Cleanup
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_gen_to_missing");
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub1_gen_to_missing");
+$node_subscriber2->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub2_gen_to_missing");
+
+#####################
+# TEST tab_nogen_to_gen
+#
+# publisher-side col 'b' is not-generated.
+# subscriber-side col 'b' is generated
+#####################
+
+# regress_sub1_nogen_to_gen: (include_generated_columns = false)
+# Confirm that col 'b' s not replicated and it will throw an error.
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_nogen_to_gen FOR TABLE tab_nogen_to_gen");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset);
+
+# regress_sub2_nogen_to_gen: (include_generated_columns = true)
+# Confirm that col 'b' s not replicated and it will throw an error.
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = true)"
+);
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" has a generated column "b" but corresponding column on source relation is not a generated column/,
+ $offset2);
+
+#Cleanup
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_nogen_to_gen");
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub1_nogen_to_gen");
+$node_subscriber2->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub2_nogen_to_gen");
+
#####################
# TEST tab_order:
#
@@ -368,7 +418,10 @@ $node_publisher->wait_for_catchup('regress_sub_misc');
$result =
$node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
-is( $result, qq(4|8|88
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
+4|8|88
5|10|110), 'replicate generated columns with different order on subscriber');
#####################
@@ -385,9 +438,9 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->wait_for_catchup('regress_sub_misc');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
-is( $result, qq(1||22
-2||44
-3||66), 'add new table to existing publication');
+is( $result, qq(1|2|22
+2|4|44
+3|6|66), 'add new table to existing publication');
#####################
# TEST tabl_alter
@@ -407,9 +460,9 @@ $node_publisher->safe_psql('postgres',
# confirmed replication now works for the subscriber nogen col
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
-is( $result, qq(1||22
-2||44
-3||66
+is( $result, qq(1|2|22
+2|4|44
+3|6|66
4|8|8
5|10|10), 'after drop generated column expression');
--
2.34.1
[application/octet-stream] v22-0001-Enable-support-for-include_generated_columns-opt.patch (101.3K, ../../CAHv8RjJsGWETA9U53iRiV2+VGtnHamEJ5PKMHUcfat269kQaSQ@mail.gmail.com/4-v22-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From c3c768bf742683506574cafdeaba7980a370f01c Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Wed, 24 Jul 2024 11:22:07 +0530
Subject: [PATCH v22 1/4] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr'
PUBLICATION regress_pub_combo WITH (include_generated_columns = true,
copy_data = false)
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 +++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 ++
contrib/test_decoding/test_decoding.c | 26 +-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 ++-
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 ++++----
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 365 +++++++++++++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 739 insertions(+), 141 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 79cd599692..3320c25a60 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3322,6 +3322,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6540,8 +6551,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..819a124c63 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ec96b5fe85..7564173bee 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4376,6 +4376,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2b02148559..de52617ded 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5167,6 +5177,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..f7b8d59413 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 024469474d..3c7e563807 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..05b83f6bec 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -14,10 +14,16 @@ my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
$node_publisher->start;
+# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
$node_subscriber->start;
+# All subscribers on this node will use parameter include_generated_columns = true
+my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$node_subscriber2->init;
+$node_subscriber2->start;
+
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
@@ -28,32 +34,184 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', with different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)"
+);
+
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+
+# tab_order:
+# publisher-side has generated cols 'b' and 'c' but
+# subscriber-side has non-generated col 'b', and generated-col 'c'
+# where columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab_alter:
+# for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_nogen_to_gen FOR TABLE tab_nogen_to_gen");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub1"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+);
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+);
+
+#####################
# Wait for initial sync of all subscriptions
+#####################
+
+# Here, copy_data = false because COPY and include_generated_columns are not
+# allowed at the same time for patch 0001.
+# And that is why all expected results on subscriber2 will be empty.
+# This limitation will be changed in patch 0002.
+
$node_subscriber->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+# gen-to-gen
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(1|21
+2|22
+3|23), 'generated columns initial sync, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+
+# gen-to-nogen
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'generated columns initial sync, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+
+# nogen-to-gen
+$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen");
+is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+
+# missing-to_gen
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is( $result, qq(), 'generated column initial sync');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(), 'unsubscribed table initial data');
+
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -62,8 +220,207 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#####################
+# TEST tab_gen_to_gen
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', using a different computation.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# regress_sub_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+#$node_publisher->wait_for_catchup('regress_pub_combo');
+$node_publisher->wait_for_catchup('regress_sub_combo');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(1|21
+2|22
+3|23
+4|24
+5|25),
+ 'confirm generated columns are NOT replicated, when include_generated_columns=false'
+);
+
+# regress_sub_combo2: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+$node_publisher->wait_for_catchup('regress_sub_combo2');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(4|24
+5|25),
+ 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+);
+
+#####################
+# TEST tab_gen_to_nogen
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# regress_sub_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub_combo');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# regress_sub_combo2: (include_generated_columns = true)
+# Confirm that col 'b' is replicated.
+$node_publisher->wait_for_catchup('regress_sub_combo2');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#####################
+# TEST tab_nogen_to_gen
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+# regress_sub_nogen_to_gen: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub_nogen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#Cleanup
+$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_nogen_to_gen");
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_nogen_to_gen");
+
+#####################
+# TEST tab_missing_to_gen
+#
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# regress_sub_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub_combo');
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|2
+2|4
+3|6
+4|8
+5|10),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# regress_sub_combo2: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub_combo2');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+#Cleanup
+$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_combo");
+$node_subscriber->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo");
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo2");
+
+#####################
+# TEST tab_order:
+#
+# publisher-side cols 'b' and 'c' are generated
+# subscriber-side col 'b' is not generated and col 'c' is generated.
+# But pub/sub table cols are in different order.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_order VALUES (4), (5)");
+
+# regress_sub_misc: (include_generated_columns = true)
+# Confirm depsite different orders replication occurs to the correct columns
+$node_publisher->wait_for_catchup('regress_sub_misc');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is( $result, qq(4|8|88
+5|10|110), 'replicate generated columns with different order on subscriber');
+
+#####################
+# TEST tab_alter
+#
+# Add new table to existing publication, then
+# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#####################
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION regress_sub_misc REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('regress_sub_misc');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66), 'add new table to existing publication');
+
+#####################
+# TEST tabl_alter
+#
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#####################
+
+# change a gencol to a nogen col
+$node_subscriber2->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN c DROP EXPRESSION");
+
+# insert some data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (4), (5)");
+
+# confirmed replication now works for the subscriber nogen col
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
+#Cleanup
+$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_misc");
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_misc");
+
+#####################
# try it with a subscriber-side trigger
+
$node_subscriber->safe_psql(
'postgres', q{
CREATE FUNCTION tab1_trigger_func() RETURNS trigger
@@ -84,7 +441,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.34.1
[application/octet-stream] v22-0004-Improve-include-generated-column-option-handling.patch (15.3K, ../../CAHv8RjJsGWETA9U53iRiV2+VGtnHamEJ5PKMHUcfat269kQaSQ@mail.gmail.com/5-v22-0004-Improve-include-generated-column-option-handling.patch)
download | inline diff:
From 923d0a3551fad01ff41c11057fc07f758bffe472 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Fri, 26 Jul 2024 12:42:42 +0530
Subject: [PATCH v22 4/4] Improve include generated column option handling by
using bms
Improve include generated column option handling by using bms.
---
src/backend/replication/logical/proto.c | 72 +++-------------
src/backend/replication/pgoutput/pgoutput.c | 95 ++++++++++++++-------
src/include/replication/logicalproto.h | 12 +--
3 files changed, 80 insertions(+), 99 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index cad1b76e7a..6b085e555c 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,12 +30,10 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -414,8 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -427,8 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -461,8 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -483,13 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -539,7 +532,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -559,8 +552,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns,
- include_generated_columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
/*
@@ -676,7 +668,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
char *relname;
@@ -698,7 +690,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns, include_generated_columns);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -775,8 +767,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -793,15 +784,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -823,15 +805,6 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -950,8 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
- bool include_generated_columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -969,15 +941,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
@@ -999,15 +962,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (!column_in_column_list(att->attnum, columns))
continue;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index c02de23743..0c4c7ac5ba 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,8 +86,7 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -165,8 +164,12 @@ typedef struct RelationSyncEntry
AttrMap *attrmap;
/*
- * Columns included in the publication, or NULL if all columns are
- * included implicitly. Note that the attnums in this bitmap are not
+ * Columns should be publicated, or NULL if all columns are included
+ * implicitly. This bitmap only considers the column list of the
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
@@ -744,13 +747,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns,
- data->include_generated_columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -764,7 +765,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns, bool include_generated_columns)
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -784,15 +785,6 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->attisdropped)
continue;
- if (att->attgenerated)
- {
- if (!include_generated_columns)
- continue;
-
- if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
- }
-
if (att->atttypid < FirstGenbkiObjectId)
continue;
@@ -806,7 +798,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -1032,6 +1024,36 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Prepare new column list bitmap. This includes all the columns of the table.
+ */
+static Bitmapset *
+prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
+ TupleDesc desc)
+{
+ Bitmapset *cols = NULL;
+ MemoryContext oldcxt = NULL;
+
+ pgoutput_ensure_entry_cxt(data, entry);
+ oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped */
+ if (att->attisdropped)
+ continue;
+
+ /* Iterate the cols until generated columns are found. */
+ cols = bms_add_member(cols, i + 1);
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+
+ return cols;
+}
+
/*
* Initialize the column list.
*/
@@ -1070,7 +1092,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
* if there are no column lists (even if other publications have a
* list).
*/
- if (!pub->alltables)
+ if (!pub->alltables || !data->include_generated_columns)
{
bool pub_no_list = true;
@@ -1091,9 +1113,10 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
Anum_pg_publication_rel_prattrs,
&pub_no_list);
+ }
/* Build the column list bitmap in the per-entry context. */
- if (!pub_no_list) /* when not null */
+ if (!pub_no_list || !data->include_generated_columns) /* when not null */
{
int i;
int nliveatts = 0;
@@ -1101,19 +1124,30 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
pgoutput_ensure_entry_cxt(data, entry);
- cols = pub_collist_to_bitmapset(cols, cfdatum,
+ if (!pub_no_list)
+ cols = pub_collist_to_bitmapset(cols, cfdatum,
entry->entry_cxt);
+ else
+ cols = prepare_all_columns_bms(data, entry, desc);
/* Get the number of live attributes. */
for (i = 0; i < desc->natts; i++)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
+ /* Skip if the attribute is dropped */
if (att->attisdropped)
continue;
-
- if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
- continue;
+ /*
+ * If column list contain generated column it will not replicate
+ * the table to the subscriber port.
+ */
+ if (att->attgenerated &&
+ (att->attgenerated != ATTRIBUTE_GENERATED_STORED ||
+ !data->include_generated_columns))
+ {
+ cols = bms_del_member(cols, i + 1);
+ }
nliveatts++;
}
@@ -1129,8 +1163,8 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
}
}
+ if (HeapTupleIsValid(cftuple))
ReleaseSysCache(cftuple);
- }
}
if (first)
@@ -1558,18 +1592,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns,
- data->include_generated_columns);
+ new_slot, data->binary, relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns,
- data->include_generated_columns);
+ data->binary, relentry->columns);
break;
default:
Assert(false);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 34ec40b07e..b9a64d9c95 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,22 +225,19 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
TupleTableSlot *newslot, bool binary,
- Bitmapset *columns,
- bool include_generated_columns);
+ Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns,
- bool include_generated_columns);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -251,8 +248,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns,
- bool include_generated_columns);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-01 08:32 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-08-01 08:32 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, Here are my review comments for patch v22-0001
All comments now are only for the TAP test.
======
src/test/subscription/t/011_generated.pl
1. I added all new code for the missing combination test case
"gen-to-missing". See nitpicks diff.
- create a separate publication for this "tab_gen_to_missing" table
because the test gives subscription errors.
- for the initial data
- for the replicated data
~~~
2. I added sub1 and sub2 subscriptions for every combo test
(previously some were absent). See nitpicks diff.
~~~
3. There was a missing test case for nogen-to-gen combination, and
after experimenting with this I am getting a bit suspicious,
Currently, it seems that if a COPY is attempted then the error would
be like this:
2024-08-01 17:16:45.110 AEST [18942] ERROR: column "b" is a generated column
2024-08-01 17:16:45.110 AEST [18942] DETAIL: Generated columns cannot
be used in COPY.
OTOH, if a COPY is not attempted (e.g. copy_data = false) then patch
0001 allows replication to happen. And the generated value of the
subscriber "b" takes precedence.
I have included these tests in the nitpicks diff of patch 0001.
Those results weren't exactly what I was expecting. That is why it is
so important to include *every* test combination in these TAP tests --
because unless we know how it works today, we won't know if we are
accidentally breaking the current behaviour with the other (0002,
0003) patches.
Please experiment in patches 0001 and 0002 using tab_nogen_to_gen more
to make sure the (new?) patch errors make sense and don't overstep by
giving ERRORs when they should not.
~~~~
Also, many other smaller issues/changes were done:
~~~
Creating tables:
nitpick - rearranged to keep all combo test SQLs in a consistent order
throughout this file
1/ gen-to-gen
2/ gen-to-nogen
3/ gen-to-missing
4/ missing-to-gen
5/ nogen-to-gen
nitpick - fixed the wrong comment for CREATE TABLE tab_nogen_to_gen.
nitpick - tweaked some CREATE TABLE comments for consistency.
nitpick - in the v22 patch many of the generated col 'b' use different
computations for every test. It makes it unnecessarily difficult to
read/review the expected results. So, I've made them all the same. Now
computation is "a * 2" on the publisher side, and "a * 22" on the
subscriber side.
~~~
Creating Publications and Subscriptions:
nitpick - added comment for all the CREATE PUBLICATION
nitpick - added comment for all the CREATE SUBSCRIPTION
nitpick - I moved the note about copy_data = false to where all the
node_subscriber2 subscriptions are created. Also, don't explicitly
refer to "patch 000" in the comment, because that will not make any
sense after getting pushed.
nitpick - I changed many subscriber names to consistently use "sub1"
or "sub2" within the name (this is the visual cue of which
node_subscriber<n> they are on). e.g.
/regress_sub_combo2/regress_sub2_combo/
~~~
Initial Sync tests:
nitpick - not sure if it is possible to do the initial data tests for
"nogen_to_gen" in the normal place. For now, it is just replaced by a
comment.
NOTE - Maybe this should be refactored later to put all the initial
data checks in one place. I'll think about this point more in the next
review.
~~~
nitpick - Changed cleanup I drop subscriptions before publications.
nitpick - remove the unnecessary blank line at the end.
======
Please see the attached diffs patch (apply it atop patch 0001) which
includes all the nipick changes mentioned above.
~~
BTW, For a quicker turnaround and less churning please consider just
posting the v23-0001 by itself instead of waiting to rebase all the
subsequent patches. When 0001 settles down some more then rebase the
others.
~~
Also, please run the indentation tool over this code ASAP.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 05b83f6..504714a 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -34,59 +34,60 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab_gen_to_gen:
# publisher-side has generated col 'b'.
# subscriber-side has generated col 'b', with different computation.
$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+# tab_gen_to_nogen:
# publisher-side has generated col 'b'.
# subscriber-side has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
-$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
-$node_subscriber2->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+# tab_gen_to_missing:
# publisher-side has generated col 'b'.
# subscriber-side col 'b' is missing.
$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
-);
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_missing (a int)"
-);
+ "CREATE TABLE tab_gen_to_missing (a int)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_missing (a int)"
-);
+ "CREATE TABLE tab_gen_to_missing (a int)");
+# tab_missing_to_gen:
# publisher-side col 'b' is missing.
-# subscriber-side col 'b' is generated.
-$node_publisher->safe_psql('postgres', "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
-# publisher-side col 'b' is missing.
-# subscriber-side col 'b' is generated.
+# tab_nogen_to_gen:
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab_missing_to_gen (a int)"
-);
+ "CREATE TABLE tab_nogen_to_gen (a int, b int)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
-);
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
-);
-
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
# tab_order:
-# publisher-side has generated cols 'b' and 'c' but
-# subscriber-side has non-generated col 'b', and generated-col 'c'
-# where columns on publisher/subscriber are in a different order
+# publisher-side has generated cols 'b' and 'c'.
+# subscriber-side has non-generated col 'b', and generated-col 'c'.
+# columns on publisher/subscriber are in a different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -107,6 +108,7 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
@@ -114,52 +116,58 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
-$node_publisher->safe_psql('postgres',
"INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+
+$node_publisher->safe_psql('postgres',
"INSERT INTO tab_order (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+# create publications
+#
+# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
+
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub1 FOR TABLE tab1");
+ "CREATE PUBLICATION regress_pub FOR TABLE tab1");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen");
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub_nogen_to_gen FOR TABLE tab_nogen_to_gen");
+ "CREATE PUBLICATION regress_pub_combo_gen_to_missing FOR TABLE tab_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_nogen_to_gen FOR TABLE tab_nogen_to_gen");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+# create subscriptions
+#
+# Note that all subscriptions created on node_subscriber2 use copy_data = false,
+# because copy_data = true with include_generated_columns is not yet supported.
+# For this reason, the expected inital data on snode_subscriber2 is always empty.
+
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub1"
+ "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub"
);
-
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+ "CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
);
-
-$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+
$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
);
-
$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
);
#####################
# Wait for initial sync of all subscriptions
#####################
-# Here, copy_data = false because COPY and include_generated_columns are not
-# allowed at the same time for patch 0001.
-# And that is why all expected results on subscriber2 will be empty.
-# This limitation will be changed in patch 0002.
-
$node_subscriber->wait_for_subscription_sync;
$node_subscriber2->wait_for_subscription_sync;
@@ -170,9 +178,9 @@ is( $result, qq(1|22
# gen-to-gen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
-is( $result, qq(1|21
-2|22
-3|23), 'generated columns initial sync, when include_generated_columns=false');
+is( $result, qq(1|22
+2|44
+3|66), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
@@ -184,18 +192,25 @@ is( $result, qq(1|
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
-# nogen-to-gen
-$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+# gen-to-missing
+# Note, node_subscriber2 is not subscribing to this yet. See later.
+$result = $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'generated columns initial sync, when include_generated_columns=false');
-# missing-to_gen
+# missing-to-gen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
-is( $result, qq(1|2
-2|4
-3|6), 'generated columns initial sync, when include_generated_columns=false');
+is( $result, qq(1|22
+2|44
+3|66), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+# nogen-to-gen
+# Note, node_subscriber is not subscribing to this yet. See later
+# Note, node_subscriber2 is not subscribing to this yet. See later
+
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
is( $result, qq(), 'generated column initial sync');
@@ -204,7 +219,6 @@ $result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is( $result, qq(), 'unsubscribed table initial data');
-
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -230,30 +244,30 @@ is( $result, qq(1|22|
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
-# regress_sub_combo: (include_generated_columns = false)
+# regress_sub1_combo: (include_generated_columns = false)
# Confirm that col 'b' is not replicated.
#$node_publisher->wait_for_catchup('regress_pub_combo');
-$node_publisher->wait_for_catchup('regress_sub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
-is( $result, qq(1|21
-2|22
-3|23
-4|24
-5|25),
- 'confirm generated columns are NOT replicated, when include_generated_columns=false'
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm generated columns are not replicated when include_generated_columns=false'
);
-# regress_sub_combo2: (include_generated_columns = true)
+# regress_sub2_combo: (include_generated_columns = true)
# Confirm that col 'b' is not replicated. We can know this because the result
# value is the subscriber-side computation (which is different from the
# publisher-side computation for this column).
-$node_publisher->wait_for_catchup('regress_sub_combo2');
+$node_publisher->wait_for_catchup('regress_sub2_combo');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
-is( $result, qq(4|24
-5|25),
- 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+is( $result, qq(4|88
+5|110),
+ 'confirm generated columns are not replicated when the subscriber-side column is also generated'
);
#####################
@@ -266,9 +280,9 @@ is( $result, qq(4|24
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
-# regress_sub_combo: (include_generated_columns = false)
+# regress_sub1_combo: (include_generated_columns = false)
# Confirm that col 'b' is not replicated.
-$node_publisher->wait_for_catchup('regress_sub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
is( $result, qq(1|
@@ -279,9 +293,9 @@ is( $result, qq(1|
'confirm generated columns are not replicated when the subscriber-side column is not generated'
);
-# regress_sub_combo2: (include_generated_columns = true)
+# regress_sub2_combo: (include_generated_columns = true)
# Confirm that col 'b' is replicated.
-$node_publisher->wait_for_catchup('regress_sub_combo2');
+$node_publisher->wait_for_catchup('regress_sub2_combo');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
is( $result, qq(4|8
@@ -290,28 +304,48 @@ is( $result, qq(4|8
);
#####################
-# TEST tab_nogen_to_gen
+# TEST tab_gen_to_missing
#
# publisher-side has generated col 'b'.
-# subscriber-side has non-generated col 'b'.
+# subscriber-side col 'b' is missing.
#####################
# insert data
-$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
-# regress_sub_nogen_to_gen: (include_generated_columns = false)
+# regress_sub1_combo_gen_to_missing: (include_generated_columns = false)
# Confirm that col 'b' is not replicated.
-$node_publisher->wait_for_catchup('regress_sub_nogen_to_gen');
+$node_publisher->wait_for_catchup('regress_sub1_combo_gen_to_missing');
$result =
- $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
-is( $result, qq(4|88
-5|110),
- 'confirm generated columns are replicated when the subscriber-side column is not generated'
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5),
+ 'missing generated column, include_generated_columns = false'
+);
+
+# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated and it will throw an error.
+my $offset2 = -s $node_subscriber2->logfile;
+
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
);
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_missing VALUES (6)");
+
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
#Cleanup
-$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_nogen_to_gen");
-$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_nogen_to_gen");
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
+$node_subscriber2->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub2_combo_gen_to_missing");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo_gen_to_missing");
#####################
# TEST tab_missing_to_gen
@@ -323,33 +357,79 @@ $node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_nogen_to_
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
-# regress_sub_combo: (include_generated_columns = false)
+# regress_sub1_combo: (include_generated_columns = false)
# Confirm that col 'b' is not replicated, but is generated as normal
-$node_publisher->wait_for_catchup('regress_sub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
-is( $result, qq(1|2
-2|4
-3|6
-4|8
-5|10),
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
'confirm when publisher col is missing, subscriber generated columns are generated as normal'
);
-# regress_sub_combo2: (include_generated_columns = true)
+# regress_sub2_combo: (include_generated_columns = true)
# Confirm that col 'b' is not replicated, but is generated as normal
-$node_publisher->wait_for_catchup('regress_sub_combo2');
+$node_publisher->wait_for_catchup('regress_sub2_combo');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
-is( $result, qq(4|8
-5|10),
+is( $result, qq(4|88
+5|110),
'confirm when publisher col is missing, subscriber generated columns are generated as normal'
);
-#Cleanup
+# cleanup
+$node_subscriber->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub1_combo");
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub2_combo");
$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_combo");
-$node_subscriber->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo");
-$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo2");
+
+#####################
+# TEST tab_nogen_to_gen
+#
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated and it will throw a COPY error.
+#
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/,
+ $offset);
+
+# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+#
+# XXX
+# when copy_data=false, no COPY error occurs.
+# the col 'b' is not replicated; the subscriber-side generated value is inserted.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (6)");
+
+$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is( $result, qq(6|132),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_combo_nogen_to_gen");
#####################
# TEST tab_order:
@@ -362,27 +442,27 @@ $node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo2");
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_order VALUES (4), (5)");
-# regress_sub_misc: (include_generated_columns = true)
-# Confirm depsite different orders replication occurs to the correct columns
-$node_publisher->wait_for_catchup('regress_sub_misc');
+# regress_sub2_misc: (include_generated_columns = true)
+# Confirm that depsite different orders replication occurs to the correct columns
+$node_publisher->wait_for_catchup('regress_sub2_misc');
$result =
$node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
is( $result, qq(4|8|88
-5|10|110), 'replicate generated columns with different order on subscriber');
+5|10|110), 'replicate generated columns with different order on the subscriber');
#####################
# TEST tab_alter
#
-# Add new table to existing publication, then
+# Add a new table to existing publication, then
# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
#####################
$node_publisher->safe_psql('postgres',
"ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
$node_subscriber2->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_sub_misc REFRESH PUBLICATION");
-$node_publisher->wait_for_catchup('regress_sub_misc');
+ "ALTER SUBSCRIPTION regress_sub2_misc REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('regress_sub2_misc');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is( $result, qq(1||22
@@ -404,7 +484,7 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_alter (a) VALUES (4), (5)");
-# confirmed replication now works for the subscriber nogen col
+# confirm that replication now works for the subscriber nogen col
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is( $result, qq(1||22
@@ -414,8 +494,8 @@ is( $result, qq(1||22
5|10|10), 'after drop generated column expression');
#Cleanup
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub2_misc");
$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_misc");
-$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_misc");
#####################
# try it with a subscriber-side trigger
Attachments:
[text/plain] PS_NITPICKS_20240801_gencols_v220001.txt (22.0K, ../../CAHut+PsJvLTXfo_GcRz+e+s_KC8nEPY5xtTt-39cKUvfs=MsJQ@mail.gmail.com/2-PS_NITPICKS_20240801_gencols_v220001.txt)
download | inline diff:
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 05b83f6..504714a 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -34,59 +34,60 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab_gen_to_gen:
# publisher-side has generated col 'b'.
# subscriber-side has generated col 'b', with different computation.
$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 10) STORED)");
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a + 20) STORED)");
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+# tab_gen_to_nogen:
# publisher-side has generated col 'b'.
# subscriber-side has non-generated col 'b'.
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
-$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
-$node_subscriber2->safe_psql('postgres', "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+# tab_gen_to_missing:
# publisher-side has generated col 'b'.
# subscriber-side col 'b' is missing.
$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
-);
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_missing (a int)"
-);
+ "CREATE TABLE tab_gen_to_missing (a int)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_gen_to_missing (a int)"
-);
+ "CREATE TABLE tab_gen_to_missing (a int)");
+# tab_missing_to_gen:
# publisher-side col 'b' is missing.
-# subscriber-side col 'b' is generated.
-$node_publisher->safe_psql('postgres', "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
-# publisher-side col 'b' is missing.
-# subscriber-side col 'b' is generated.
+# tab_nogen_to_gen:
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab_missing_to_gen (a int)"
-);
+ "CREATE TABLE tab_nogen_to_gen (a int, b int)");
$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
-);
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
$node_subscriber2->safe_psql('postgres',
- "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
-);
-
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)");
# tab_order:
-# publisher-side has generated cols 'b' and 'c' but
-# subscriber-side has non-generated col 'b', and generated-col 'c'
-# where columns on publisher/subscriber are in a different order
+# publisher-side has generated cols 'b' and 'c'.
+# subscriber-side has non-generated col 'b', and generated-col 'c'.
+# columns on publisher/subscriber are in a different order
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
);
@@ -107,6 +108,7 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
@@ -114,52 +116,58 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
-$node_publisher->safe_psql('postgres',
"INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+
+$node_publisher->safe_psql('postgres',
"INSERT INTO tab_order (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+# create publications
+#
+# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
+
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub1 FOR TABLE tab1");
+ "CREATE PUBLICATION regress_pub FOR TABLE tab1");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen");
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub_nogen_to_gen FOR TABLE tab_nogen_to_gen");
+ "CREATE PUBLICATION regress_pub_combo_gen_to_missing FOR TABLE tab_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_nogen_to_gen FOR TABLE tab_nogen_to_gen");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+# create subscriptions
+#
+# Note that all subscriptions created on node_subscriber2 use copy_data = false,
+# because copy_data = true with include_generated_columns is not yet supported.
+# For this reason, the expected inital data on snode_subscriber2 is always empty.
+
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub1"
+ "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub"
);
-
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+ "CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
);
-
-$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+
$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
);
-
$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+ "CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
);
#####################
# Wait for initial sync of all subscriptions
#####################
-# Here, copy_data = false because COPY and include_generated_columns are not
-# allowed at the same time for patch 0001.
-# And that is why all expected results on subscriber2 will be empty.
-# This limitation will be changed in patch 0002.
-
$node_subscriber->wait_for_subscription_sync;
$node_subscriber2->wait_for_subscription_sync;
@@ -170,9 +178,9 @@ is( $result, qq(1|22
# gen-to-gen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
-is( $result, qq(1|21
-2|22
-3|23), 'generated columns initial sync, when include_generated_columns=false');
+is( $result, qq(1|22
+2|44
+3|66), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
@@ -184,18 +192,25 @@ is( $result, qq(1|
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen");
is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
-# nogen-to-gen
-$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen");
-is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+# gen-to-missing
+# Note, node_subscriber2 is not subscribing to this yet. See later.
+$result = $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'generated columns initial sync, when include_generated_columns=false');
-# missing-to_gen
+# missing-to-gen
$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
-is( $result, qq(1|2
-2|4
-3|6), 'generated columns initial sync, when include_generated_columns=false');
+is( $result, qq(1|22
+2|44
+3|66), 'generated columns initial sync, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen");
is( $result, qq(), 'generated columns initial sync, when include_generated_columns=true');
+# nogen-to-gen
+# Note, node_subscriber is not subscribing to this yet. See later
+# Note, node_subscriber2 is not subscribing to this yet. See later
+
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
is( $result, qq(), 'generated column initial sync');
@@ -204,7 +219,6 @@ $result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is( $result, qq(), 'unsubscribed table initial data');
-
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
@@ -230,30 +244,30 @@ is( $result, qq(1|22|
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
-# regress_sub_combo: (include_generated_columns = false)
+# regress_sub1_combo: (include_generated_columns = false)
# Confirm that col 'b' is not replicated.
#$node_publisher->wait_for_catchup('regress_pub_combo');
-$node_publisher->wait_for_catchup('regress_sub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
-is( $result, qq(1|21
-2|22
-3|23
-4|24
-5|25),
- 'confirm generated columns are NOT replicated, when include_generated_columns=false'
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm generated columns are not replicated when include_generated_columns=false'
);
-# regress_sub_combo2: (include_generated_columns = true)
+# regress_sub2_combo: (include_generated_columns = true)
# Confirm that col 'b' is not replicated. We can know this because the result
# value is the subscriber-side computation (which is different from the
# publisher-side computation for this column).
-$node_publisher->wait_for_catchup('regress_sub_combo2');
+$node_publisher->wait_for_catchup('regress_sub2_combo');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
-is( $result, qq(4|24
-5|25),
- 'confirm generated columns are NOT replicated when the subscriber-side column is also generated'
+is( $result, qq(4|88
+5|110),
+ 'confirm generated columns are not replicated when the subscriber-side column is also generated'
);
#####################
@@ -266,9 +280,9 @@ is( $result, qq(4|24
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
-# regress_sub_combo: (include_generated_columns = false)
+# regress_sub1_combo: (include_generated_columns = false)
# Confirm that col 'b' is not replicated.
-$node_publisher->wait_for_catchup('regress_sub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
is( $result, qq(1|
@@ -279,9 +293,9 @@ is( $result, qq(1|
'confirm generated columns are not replicated when the subscriber-side column is not generated'
);
-# regress_sub_combo2: (include_generated_columns = true)
+# regress_sub2_combo: (include_generated_columns = true)
# Confirm that col 'b' is replicated.
-$node_publisher->wait_for_catchup('regress_sub_combo2');
+$node_publisher->wait_for_catchup('regress_sub2_combo');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
is( $result, qq(4|8
@@ -290,28 +304,48 @@ is( $result, qq(4|8
);
#####################
-# TEST tab_nogen_to_gen
+# TEST tab_gen_to_missing
#
# publisher-side has generated col 'b'.
-# subscriber-side has non-generated col 'b'.
+# subscriber-side col 'b' is missing.
#####################
# insert data
-$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
-# regress_sub_nogen_to_gen: (include_generated_columns = false)
+# regress_sub1_combo_gen_to_missing: (include_generated_columns = false)
# Confirm that col 'b' is not replicated.
-$node_publisher->wait_for_catchup('regress_sub_nogen_to_gen');
+$node_publisher->wait_for_catchup('regress_sub1_combo_gen_to_missing');
$result =
- $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
-is( $result, qq(4|88
-5|110),
- 'confirm generated columns are replicated when the subscriber-side column is not generated'
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5),
+ 'missing generated column, include_generated_columns = false'
+);
+
+# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated and it will throw an error.
+my $offset2 = -s $node_subscriber2->logfile;
+
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
);
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_gen_to_missing VALUES (6)");
+
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
#Cleanup
-$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_nogen_to_gen");
-$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_nogen_to_gen");
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
+$node_subscriber2->safe_psql('postgres', "DROP SUBSCRIPTION regress_sub2_combo_gen_to_missing");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo_gen_to_missing");
#####################
# TEST tab_missing_to_gen
@@ -323,33 +357,79 @@ $node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_nogen_to_
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
-# regress_sub_combo: (include_generated_columns = false)
+# regress_sub1_combo: (include_generated_columns = false)
# Confirm that col 'b' is not replicated, but is generated as normal
-$node_publisher->wait_for_catchup('regress_sub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
-is( $result, qq(1|2
-2|4
-3|6
-4|8
-5|10),
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
'confirm when publisher col is missing, subscriber generated columns are generated as normal'
);
-# regress_sub_combo2: (include_generated_columns = true)
+# regress_sub2_combo: (include_generated_columns = true)
# Confirm that col 'b' is not replicated, but is generated as normal
-$node_publisher->wait_for_catchup('regress_sub_combo2');
+$node_publisher->wait_for_catchup('regress_sub2_combo');
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
-is( $result, qq(4|8
-5|10),
+is( $result, qq(4|88
+5|110),
'confirm when publisher col is missing, subscriber generated columns are generated as normal'
);
-#Cleanup
+# cleanup
+$node_subscriber->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub1_combo");
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub2_combo");
$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_combo");
-$node_subscriber->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo");
-$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo2");
+
+#####################
+# TEST tab_nogen_to_gen
+#
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated and it will throw a COPY error.
+#
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/,
+ $offset);
+
+# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+#
+# XXX
+# when copy_data=false, no COPY error occurs.
+# the col 'b' is not replicated; the subscriber-side generated value is inserted.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_nogen_to_gen VALUES (6)");
+
+$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is( $result, qq(6|132),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_combo_nogen_to_gen");
#####################
# TEST tab_order:
@@ -362,27 +442,27 @@ $node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_combo2");
# insert data
$node_publisher->safe_psql('postgres', "INSERT INTO tab_order VALUES (4), (5)");
-# regress_sub_misc: (include_generated_columns = true)
-# Confirm depsite different orders replication occurs to the correct columns
-$node_publisher->wait_for_catchup('regress_sub_misc');
+# regress_sub2_misc: (include_generated_columns = true)
+# Confirm that depsite different orders replication occurs to the correct columns
+$node_publisher->wait_for_catchup('regress_sub2_misc');
$result =
$node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
is( $result, qq(4|8|88
-5|10|110), 'replicate generated columns with different order on subscriber');
+5|10|110), 'replicate generated columns with different order on the subscriber');
#####################
# TEST tab_alter
#
-# Add new table to existing publication, then
+# Add a new table to existing publication, then
# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
#####################
$node_publisher->safe_psql('postgres',
"ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
$node_subscriber2->safe_psql('postgres',
- "ALTER SUBSCRIPTION regress_sub_misc REFRESH PUBLICATION");
-$node_publisher->wait_for_catchup('regress_sub_misc');
+ "ALTER SUBSCRIPTION regress_sub2_misc REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('regress_sub2_misc');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is( $result, qq(1||22
@@ -404,7 +484,7 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_alter (a) VALUES (4), (5)");
-# confirmed replication now works for the subscriber nogen col
+# confirm that replication now works for the subscriber nogen col
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is( $result, qq(1||22
@@ -414,8 +494,8 @@ is( $result, qq(1||22
5|10|10), 'after drop generated column expression');
#Cleanup
+$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub2_misc");
$node_publisher->safe_psql('postgres',"DROP PUBLICATION regress_pub_misc");
-$node_subscriber2->safe_psql('postgres',"DROP SUBSCRIPTION regress_sub_misc");
#####################
# try it with a subscriber-side trigger
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-02 04:26 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Shubham Khanna @ 2024-08-02 04:26 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Aug 1, 2024 at 2:02 PM Peter Smith <[email protected]> wrote:
>
> Hi, Here are my review comments for patch v22-0001
>
> All comments now are only for the TAP test.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> 1. I added all new code for the missing combination test case
> "gen-to-missing". See nitpicks diff.
> - create a separate publication for this "tab_gen_to_missing" table
> because the test gives subscription errors.
> - for the initial data
> - for the replicated data
>
> ~~~
>
> 2. I added sub1 and sub2 subscriptions for every combo test
> (previously some were absent). See nitpicks diff.
>
> ~~~
>
> 3. There was a missing test case for nogen-to-gen combination, and
> after experimenting with this I am getting a bit suspicious,
>
> Currently, it seems that if a COPY is attempted then the error would
> be like this:
> 2024-08-01 17:16:45.110 AEST [18942] ERROR: column "b" is a generated column
> 2024-08-01 17:16:45.110 AEST [18942] DETAIL: Generated columns cannot
> be used in COPY.
>
> OTOH, if a COPY is not attempted (e.g. copy_data = false) then patch
> 0001 allows replication to happen. And the generated value of the
> subscriber "b" takes precedence.
>
> I have included these tests in the nitpicks diff of patch 0001.
>
> Those results weren't exactly what I was expecting. That is why it is
> so important to include *every* test combination in these TAP tests --
> because unless we know how it works today, we won't know if we are
> accidentally breaking the current behaviour with the other (0002,
> 0003) patches.
>
> Please experiment in patches 0001 and 0002 using tab_nogen_to_gen more
> to make sure the (new?) patch errors make sense and don't overstep by
> giving ERRORs when they should not.
>
> ~~~~
>
> Also, many other smaller issues/changes were done:
>
> ~~~
>
> Creating tables:
>
> nitpick - rearranged to keep all combo test SQLs in a consistent order
> throughout this file
> 1/ gen-to-gen
> 2/ gen-to-nogen
> 3/ gen-to-missing
> 4/ missing-to-gen
> 5/ nogen-to-gen
>
> nitpick - fixed the wrong comment for CREATE TABLE tab_nogen_to_gen.
>
> nitpick - tweaked some CREATE TABLE comments for consistency.
>
> nitpick - in the v22 patch many of the generated col 'b' use different
> computations for every test. It makes it unnecessarily difficult to
> read/review the expected results. So, I've made them all the same. Now
> computation is "a * 2" on the publisher side, and "a * 22" on the
> subscriber side.
>
> ~~~
>
> Creating Publications and Subscriptions:
>
>
> nitpick - added comment for all the CREATE PUBLICATION
>
> nitpick - added comment for all the CREATE SUBSCRIPTION
>
> nitpick - I moved the note about copy_data = false to where all the
> node_subscriber2 subscriptions are created. Also, don't explicitly
> refer to "patch 000" in the comment, because that will not make any
> sense after getting pushed.
>
> nitpick - I changed many subscriber names to consistently use "sub1"
> or "sub2" within the name (this is the visual cue of which
> node_subscriber<n> they are on). e.g.
> /regress_sub_combo2/regress_sub2_combo/
>
> ~~~
>
> Initial Sync tests:
>
> nitpick - not sure if it is possible to do the initial data tests for
> "nogen_to_gen" in the normal place. For now, it is just replaced by a
> comment.
> NOTE - Maybe this should be refactored later to put all the initial
> data checks in one place. I'll think about this point more in the next
> review.
>
> ~~~
>
> nitpick - Changed cleanup I drop subscriptions before publications.
>
> nitpick - remove the unnecessary blank line at the end.
>
> ======
>
> Please see the attached diffs patch (apply it atop patch 0001) which
> includes all the nipick changes mentioned above.
>
> ~~
>
> BTW, For a quicker turnaround and less churning please consider just
> posting the v23-0001 by itself instead of waiting to rebase all the
> subsequent patches. When 0001 settles down some more then rebase the
> others.
>
> ~~
>
> Also, please run the indentation tool over this code ASAP.
>
I have fixed all the comments. The attached Patch(v23-0001) contains
all the changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v23-0001-Enable-support-for-include_generated_columns-opt.patch (104.7K, ../../CAHv8RjLwBnjLrFy5En87ezpSEMj94o=BP_4x3jmQP6OgEz5EOA@mail.gmail.com/2-v23-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From aa22a4d22c488cdd2b7adf1a7e5ad6be4609e2aa Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Wed, 24 Jul 2024 11:22:07 +0530
Subject: [PATCH v23] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr'
PUBLICATION regress_pub_combo WITH (include_generated_columns = true,
copy_data = false)
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +
contrib/test_decoding/test_decoding.c | 26 +-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++---
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 492 +++++++++++++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 866 insertions(+), 141 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 79cd599692..3320c25a60 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3322,6 +3322,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6540,8 +6551,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..819a124c63 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 6dc54c7283..f40f61ed7a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4376,6 +4376,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 0d02516273..ab3df66cce 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4800,6 +4800,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4872,11 +4873,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4915,6 +4922,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4961,6 +4969,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5207,6 +5217,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..28752ade7e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 024469474d..3c7e563807 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..0b596b7dd8 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -14,10 +14,16 @@ my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
$node_publisher->start;
+# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
$node_subscriber->start;
+# All subscribers on this node will use parameter include_generated_columns = true
+my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$node_subscriber2->init;
+$node_subscriber2->start;
+
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
@@ -28,32 +34,222 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab_gen_to_gen:
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', with different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_gen_to_nogen:
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+
+# tab_gen_to_missing:
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+
+# tab_missing_to_gen:
+# publisher-side col 'b' is missing.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_nogen_to_gen:
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_order:
+# publisher-side has generated cols 'b' and 'c'.
+# subscriber-side has non-generated col 'b', and generated-col 'c'.
+# columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab_alter:
+# for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+
+# create publications
+#
+# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_gen_to_missing FOR TABLE tab_gen_to_missing"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_nogen_to_gen FOR TABLE tab_nogen_to_gen"
+);
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+
+# create subscriptions
+#
+# Note that all subscriptions created on node_subscriber2 use copy_data = false,
+# because copy_data = true with include_generated_columns is not yet supported.
+# For this reason, the expected inital data on snode_subscriber2 is always empty.
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+);
+
+#####################
# Wait for initial sync of all subscriptions
+#####################
+
$node_subscriber->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+# gen-to-gen
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'generated columns initial sync, when include_generated_columns=false'
+);
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is($result, qq(),
+ 'generated columns initial sync, when include_generated_columns=true');
+
+# gen-to-nogen
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'generated columns initial sync, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is($result, qq(),
+ 'generated columns initial sync, when include_generated_columns=true');
+
+# gen-to-missing
+# Note, node_subscriber2 is not subscribing to this yet. See later.
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'generated columns initial sync, when include_generated_columns=false');
+
+# missing-to-gen
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'generated columns initial sync, when include_generated_columns=false'
+);
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is($result, qq(),
+ 'generated columns initial sync, when include_generated_columns=true');
+
+# nogen-to-gen
+# Note, node_subscriber is not subscribing to this yet. See later
+# Note, node_subscriber2 is not subscribing to this yet. See later
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is($result, qq(), 'generated column initial sync');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is($result, qq(), 'unsubscribed table initial data');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -62,8 +258,296 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#####################
+# TEST tab_gen_to_gen
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', using a different computation.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+#$node_publisher->wait_for_catchup('regress_pub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm generated columns are not replicated when include_generated_columns=false'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm generated columns are not replicated when the subscriber-side column is also generated'
+);
+
+#####################
+# TEST tab_gen_to_nogen
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is replicated.
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#####################
+# TEST tab_gen_to_missing
+#
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
+
+# regress_sub1_combo_gen_to_missing: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo_gen_to_missing');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5),
+ 'missing generated column, include_generated_columns = false');
+
+# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated and it will throw an error.
+my $offset2 = -s $node_subscriber2->logfile;
+
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (6)");
+
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
+#Cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_gen_to_missing");
+
+#####################
+# TEST tab_missing_to_gen
+#
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo");
+
+#####################
+# TEST tab_nogen_to_gen
+#
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated and it will throw a COPY error.
+#
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/, $offset);
+
+# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+#
+# XXX
+# when copy_data=false, no COPY error occurs.
+# the col 'b' is not replicated; the subscriber-side generated value is inserted.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (6)");
+
+$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is($result, qq(6|132),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_nogen_to_gen");
+
+#####################
+# TEST tab_order:
+#
+# publisher-side cols 'b' and 'c' are generated
+# subscriber-side col 'b' is not generated and col 'c' is generated.
+# But pub/sub table cols are in different order.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order VALUES (4), (5)");
+
+# regress_sub2_misc: (include_generated_columns = true)
+# Confirm that depsite different orders replication occurs to the correct columns
+$node_publisher->wait_for_catchup('regress_sub2_misc');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is( $result, qq(4|8|88
+5|10|110),
+ 'replicate generated columns with different order on the subscriber');
+
+#####################
+# TEST tab_alter
+#
+# Add a new table to existing publication, then
+# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#####################
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION regress_sub2_misc REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('regress_sub2_misc');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66), 'add new table to existing publication');
+
+#####################
+# TEST tabl_alter
+#
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#####################
+
+# change a gencol to a nogen col
+$node_subscriber2->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN c DROP EXPRESSION");
+
+# insert some data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (4), (5)");
+
+# confirm that replication now works for the subscriber nogen col
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_misc");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_misc");
+
+#####################
# try it with a subscriber-side trigger
+
$node_subscriber->safe_psql(
'postgres', q{
CREATE FUNCTION tab1_trigger_func() RETURNS trigger
@@ -84,7 +568,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-05 02:40 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Peter Smith @ 2024-08-05 02:40 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubhab.
Here are some more review comments for the v23-0001.
======
011_generated.pl b/src/test/subscription/t/011_generated.pl
nitpick - renamed /regress_pub/regress_pub_tab1/ and
/regress_sub1/regress_sub1_tab1/
nitpick - typo /inital data /initial data/
nitpick - typo /snode_subscriber2/node_subscriber2/
nitpick - tweak the combo initial sync comments and messages
nitpick - /#Cleanup/# cleanup/
nitpick - tweak all the combo normal replication comments
nitpick - removed blank line at the end
~~~
1. Refactor tab_gen_to_missing initial sync tests.
I moved the tab_gen_to_missing initial sync for node_subscriber2 to be
back where all the other initial sync tests are done.
See the nitpicks patch file.
~~~
2. Refactor tab_nogen_to_gen initial sync tests
I moved all the tab_nogen_to_gen initial sync tests back to where the
other initial sync tests are done.
See the nitpicks patch file.
~~~
3. Added another test case:
Because the (current PG17) nogen-to-gen initial sync test case (with
copy_data=true) gives an ERROR, I have added another combination to
cover normal replication (e.g. using copy_data=false).
See the nitpicks patch file.
(This has exposed an inconsistency which IMO might be a PG17 bug. I
have included TAP test comments about this, and plan to post a
separate thread for it later).
~
4. GUC
Moving and adding more CREATE SUBSCRIPTION exceeded some default GUCs,
so extra configuration was needed.
See the nitpick patch file.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 0b596b7..2be06c6 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -12,16 +12,25 @@ use Test::More;
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ "max_wal_senders = 20
+ max_replication_slots = 20");
$node_publisher->start;
# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber->start;
# All subscribers on this node will use parameter include_generated_columns = true
my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
$node_subscriber2->init;
+$node_subscriber2->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber2->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -139,7 +148,7 @@ $node_publisher->safe_psql('postgres',
# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub FOR TABLE tab1");
+ "CREATE PUBLICATION regress_pub_tab1 FOR TABLE tab1");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen"
);
@@ -157,10 +166,10 @@ $node_publisher->safe_psql('postgres',
#
# Note that all subscriptions created on node_subscriber2 use copy_data = false,
# because copy_data = true with include_generated_columns is not yet supported.
-# For this reason, the expected inital data on snode_subscriber2 is always empty.
+# For this reason, the expected inital data on node_subscriber2 is always empty.
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub"
+ "CREATE SUBSCRIPTION regress_sub1_tab1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_tab1"
);
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
@@ -168,11 +177,18 @@ $node_subscriber->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+# Note, regress_sub1_combo_nogen_to_gen is not created here due to expected errors. See later.
$node_subscriber2->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
);
$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
);
@@ -188,57 +204,82 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
-# gen-to-gen
+#####################
+# TEST tab_gen_to_gen initial sync
+#####################
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
is( $result, qq(1|22
2|44
-3|66), 'generated columns initial sync, when include_generated_columns=false'
+3|66), 'tab_gen_to_gen initial sync, when include_generated_columns=false'
);
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
is($result, qq(),
- 'generated columns initial sync, when include_generated_columns=true');
+ 'tab_gen_to_gen initial sync, when include_generated_columns=true');
-# gen-to-nogen
+#####################
+# TEST tab_gen_to_nogen initial sync
+#####################
$result = $node_subscriber->safe_psql('postgres',
"SELECT a, b FROM tab_gen_to_nogen");
is( $result, qq(1|
2|
-3|), 'generated columns initial sync, when include_generated_columns=false');
+3|), 'tab_gen_to_nogen, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b FROM tab_gen_to_nogen");
is($result, qq(),
- 'generated columns initial sync, when include_generated_columns=true');
+ 'tab_gen_to_nogen initial sync, when include_generated_columns=true');
-# gen-to-missing
-# Note, node_subscriber2 is not subscribing to this yet. See later.
+#####################
+# TEST tab_gen_to_missing initial sync
+#####################
$result =
$node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
is( $result, qq(1
2
-3), 'generated columns initial sync, when include_generated_columns=false');
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+# Note, the following is expected to work only because copy_data = false
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(), 'tab_gen_to_missing initial sync, when include_generated_columns=true');
-# missing-to-gen
+#####################
+# TEST tab_missing_to_gen initial sync
+#####################
$result = $node_subscriber->safe_psql('postgres',
"SELECT a, b FROM tab_missing_to_gen");
is( $result, qq(1|22
2|44
-3|66), 'generated columns initial sync, when include_generated_columns=false'
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
);
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b FROM tab_missing_to_gen");
is($result, qq(),
- 'generated columns initial sync, when include_generated_columns=true');
+ 'tab_missing_to_gen initial sync, when include_generated_columns=true');
-# nogen-to-gen
-# Note, node_subscriber is not subscribing to this yet. See later
-# Note, node_subscriber2 is not subscribing to this yet. See later
+#####################
+# TEST tab_nogen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/, $offset);
+# Note, the following is expected to work only because copy_data = false
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen");
+is($result, qq(),
+ 'tab_nogen_to_gen initial sync, when include_generated_columns=true');
+# tab_order:
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
is($result, qq(), 'generated column initial sync');
+# tab_alter:
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is($result, qq(), 'unsubscribed table initial data');
@@ -249,7 +290,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('regress_sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -259,7 +300,7 @@ is( $result, qq(1|22|
6|132|), 'generated columns replicated');
#####################
-# TEST tab_gen_to_gen
+# TEST tab_gen_to_gen replication
#
# publisher-side has generated col 'b'.
# subscriber-side has generated col 'b', using a different computation.
@@ -298,7 +339,7 @@ is( $result, qq(4|88
);
#####################
-# TEST tab_gen_to_nogen
+# TEST tab_gen_to_nogen replication
#
# publisher-side has generated col 'b'.
# subscriber-side has non-generated col 'b'.
@@ -334,7 +375,7 @@ is( $result, qq(4|8
);
#####################
-# TEST tab_gen_to_missing
+# TEST tab_gen_to_missing replication
#
# publisher-side has generated col 'b'.
# subscriber-side col 'b' is missing.
@@ -360,21 +401,11 @@ is( $result, qq(1
# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
# Confirm that col 'b' is not replicated and it will throw an error.
my $offset2 = -s $node_subscriber2->logfile;
-
-# The subscription is created here, because it causes the tablesync worker to restart repetitively.
-$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
-);
-
-# insert data
-$node_publisher->safe_psql('postgres',
- "INSERT INTO tab_gen_to_missing VALUES (6)");
-
$node_subscriber2->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
$offset2);
-#Cleanup
+# cleanup
$node_subscriber->safe_psql('postgres',
"DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
$node_subscriber2->safe_psql('postgres',
@@ -383,7 +414,7 @@ $node_publisher->safe_psql('postgres',
"DROP PUBLICATION regress_pub_combo_gen_to_missing");
#####################
-# TEST tab_missing_to_gen
+# TEST tab_missing_to_gen replication
#
# publisher-side col 'b' is missing.
# subscriber-side col 'b' is generated.
@@ -426,57 +457,62 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo");
#####################
-# TEST tab_nogen_to_gen
+# TEST tab_nogen_to_gen replication
#
# publisher-side has non-generated col 'b'.
# subscriber-side has generated col 'b'.
#####################
+# When copy_data=true a COPY error occurred. Try again but with copy_data=false.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false, copy_data = false)"
+);
+
# insert data
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
-# Confirm that col 'b' is not replicated and it will throw a COPY error.
#
-# The subscription is created here, because it causes the tablesync worker to restart repetitively.
-my $offset = -s $node_subscriber->logfile;
-$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+# XXX
+# The test below shows that current PG17 behavior does not give an error,
+# But this conflicts with the copy_data=true behavior so it might be a PG17 bug.
+# Needs more study.
+$node_publisher->wait_for_catchup('regress_sub1_combo_nogen_to_gen_nocopy');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is($result, qq(4|88
+5|110),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
);
-$node_subscriber->wait_for_log(
- qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/, $offset);
# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
#
# XXX
-# when copy_data=false, no COPY error occurs.
-# the col 'b' is not replicated; the subscriber-side generated value is inserted.
-$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
-);
-
-# insert data
-$node_publisher->safe_psql('postgres',
- "INSERT INTO tab_nogen_to_gen VALUES (6)");
-
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres',
"SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
-is($result, qq(6|132),
+is($result, qq(4|88
+5|110),
'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
);
# cleanup
-
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy");
$node_subscriber2->safe_psql('postgres',
"DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
$node_publisher->safe_psql('postgres',
"DROP PUBLICATION regress_pub_combo_nogen_to_gen");
#####################
-# TEST tab_order:
+# TEST tab_order replication
#
# publisher-side cols 'b' and 'c' are generated
# subscriber-side col 'b' is not generated and col 'c' is generated.
@@ -498,7 +534,7 @@ is( $result, qq(4|8|88
'replicate generated columns with different order on the subscriber');
#####################
-# TEST tab_alter
+# TEST tab_alter replication
#
# Add a new table to existing publication, then
# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
@@ -516,7 +552,7 @@ is( $result, qq(1||22
3||66), 'add new table to existing publication');
#####################
-# TEST tabl_alter
+# TEST tab_alter
#
# Drop the generated column's expression on subscriber side.
# This changes the generated column into a non-generated column.
@@ -547,7 +583,6 @@ $node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_misc");
#####################
# try it with a subscriber-side trigger
-
$node_subscriber->safe_psql(
'postgres', q{
CREATE FUNCTION tab1_trigger_func() RETURNS trigger
@@ -568,7 +603,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('regress_sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
Attachments:
[text/plain] PS_NITPICKS_20240805_gencols_230001.txt (14.6K, ../../CAHut+PvtT8fKOfvVYr4vANx_fr92vedas+ZRbQxvMC097rks6w@mail.gmail.com/2-PS_NITPICKS_20240805_gencols_230001.txt)
download | inline diff:
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 0b596b7..2be06c6 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -12,16 +12,25 @@ use Test::More;
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ "max_wal_senders = 20
+ max_replication_slots = 20");
$node_publisher->start;
# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber->start;
# All subscribers on this node will use parameter include_generated_columns = true
my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
$node_subscriber2->init;
+$node_subscriber2->append_conf('postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber2->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -139,7 +148,7 @@ $node_publisher->safe_psql('postgres',
# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION regress_pub FOR TABLE tab1");
+ "CREATE PUBLICATION regress_pub_tab1 FOR TABLE tab1");
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen"
);
@@ -157,10 +166,10 @@ $node_publisher->safe_psql('postgres',
#
# Note that all subscriptions created on node_subscriber2 use copy_data = false,
# because copy_data = true with include_generated_columns is not yet supported.
-# For this reason, the expected inital data on snode_subscriber2 is always empty.
+# For this reason, the expected inital data on node_subscriber2 is always empty.
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub"
+ "CREATE SUBSCRIPTION regress_sub1_tab1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_tab1"
);
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
@@ -168,11 +177,18 @@ $node_subscriber->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+# Note, regress_sub1_combo_nogen_to_gen is not created here due to expected errors. See later.
$node_subscriber2->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
);
$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
"CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
);
@@ -188,57 +204,82 @@ is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
-# gen-to-gen
+#####################
+# TEST tab_gen_to_gen initial sync
+#####################
$result =
$node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
is( $result, qq(1|22
2|44
-3|66), 'generated columns initial sync, when include_generated_columns=false'
+3|66), 'tab_gen_to_gen initial sync, when include_generated_columns=false'
);
$result =
$node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
is($result, qq(),
- 'generated columns initial sync, when include_generated_columns=true');
+ 'tab_gen_to_gen initial sync, when include_generated_columns=true');
-# gen-to-nogen
+#####################
+# TEST tab_gen_to_nogen initial sync
+#####################
$result = $node_subscriber->safe_psql('postgres',
"SELECT a, b FROM tab_gen_to_nogen");
is( $result, qq(1|
2|
-3|), 'generated columns initial sync, when include_generated_columns=false');
+3|), 'tab_gen_to_nogen, when include_generated_columns=false');
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b FROM tab_gen_to_nogen");
is($result, qq(),
- 'generated columns initial sync, when include_generated_columns=true');
+ 'tab_gen_to_nogen initial sync, when include_generated_columns=true');
-# gen-to-missing
-# Note, node_subscriber2 is not subscribing to this yet. See later.
+#####################
+# TEST tab_gen_to_missing initial sync
+#####################
$result =
$node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
is( $result, qq(1
2
-3), 'generated columns initial sync, when include_generated_columns=false');
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+# Note, the following is expected to work only because copy_data = false
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(), 'tab_gen_to_missing initial sync, when include_generated_columns=true');
-# missing-to-gen
+#####################
+# TEST tab_missing_to_gen initial sync
+#####################
$result = $node_subscriber->safe_psql('postgres',
"SELECT a, b FROM tab_missing_to_gen");
is( $result, qq(1|22
2|44
-3|66), 'generated columns initial sync, when include_generated_columns=false'
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
);
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b FROM tab_missing_to_gen");
is($result, qq(),
- 'generated columns initial sync, when include_generated_columns=true');
+ 'tab_missing_to_gen initial sync, when include_generated_columns=true');
-# nogen-to-gen
-# Note, node_subscriber is not subscribing to this yet. See later
-# Note, node_subscriber2 is not subscribing to this yet. See later
+#####################
+# TEST tab_nogen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/, $offset);
+# Note, the following is expected to work only because copy_data = false
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen");
+is($result, qq(),
+ 'tab_nogen_to_gen initial sync, when include_generated_columns=true');
+# tab_order:
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_order ORDER BY a");
is($result, qq(), 'generated column initial sync');
+# tab_alter:
$result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab_alter ORDER BY a");
is($result, qq(), 'unsubscribed table initial data');
@@ -249,7 +290,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('regress_sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -259,7 +300,7 @@ is( $result, qq(1|22|
6|132|), 'generated columns replicated');
#####################
-# TEST tab_gen_to_gen
+# TEST tab_gen_to_gen replication
#
# publisher-side has generated col 'b'.
# subscriber-side has generated col 'b', using a different computation.
@@ -298,7 +339,7 @@ is( $result, qq(4|88
);
#####################
-# TEST tab_gen_to_nogen
+# TEST tab_gen_to_nogen replication
#
# publisher-side has generated col 'b'.
# subscriber-side has non-generated col 'b'.
@@ -334,7 +375,7 @@ is( $result, qq(4|8
);
#####################
-# TEST tab_gen_to_missing
+# TEST tab_gen_to_missing replication
#
# publisher-side has generated col 'b'.
# subscriber-side col 'b' is missing.
@@ -360,21 +401,11 @@ is( $result, qq(1
# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
# Confirm that col 'b' is not replicated and it will throw an error.
my $offset2 = -s $node_subscriber2->logfile;
-
-# The subscription is created here, because it causes the tablesync worker to restart repetitively.
-$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
-);
-
-# insert data
-$node_publisher->safe_psql('postgres',
- "INSERT INTO tab_gen_to_missing VALUES (6)");
-
$node_subscriber2->wait_for_log(
qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
$offset2);
-#Cleanup
+# cleanup
$node_subscriber->safe_psql('postgres',
"DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
$node_subscriber2->safe_psql('postgres',
@@ -383,7 +414,7 @@ $node_publisher->safe_psql('postgres',
"DROP PUBLICATION regress_pub_combo_gen_to_missing");
#####################
-# TEST tab_missing_to_gen
+# TEST tab_missing_to_gen replication
#
# publisher-side col 'b' is missing.
# subscriber-side col 'b' is generated.
@@ -426,57 +457,62 @@ $node_subscriber2->safe_psql('postgres',
$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo");
#####################
-# TEST tab_nogen_to_gen
+# TEST tab_nogen_to_gen replication
#
# publisher-side has non-generated col 'b'.
# subscriber-side has generated col 'b'.
#####################
+# When copy_data=true a COPY error occurred. Try again but with copy_data=false.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false, copy_data = false)"
+);
+
# insert data
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
-# Confirm that col 'b' is not replicated and it will throw a COPY error.
#
-# The subscription is created here, because it causes the tablesync worker to restart repetitively.
-my $offset = -s $node_subscriber->logfile;
-$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+# XXX
+# The test below shows that current PG17 behavior does not give an error,
+# But this conflicts with the copy_data=true behavior so it might be a PG17 bug.
+# Needs more study.
+$node_publisher->wait_for_catchup('regress_sub1_combo_nogen_to_gen_nocopy');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is($result, qq(4|88
+5|110),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
);
-$node_subscriber->wait_for_log(
- qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/, $offset);
# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
#
# XXX
-# when copy_data=false, no COPY error occurs.
-# the col 'b' is not replicated; the subscriber-side generated value is inserted.
-$node_subscriber2->safe_psql('postgres',
- "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
-);
-
-# insert data
-$node_publisher->safe_psql('postgres',
- "INSERT INTO tab_nogen_to_gen VALUES (6)");
-
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres',
"SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
-is($result, qq(6|132),
+is($result, qq(4|88
+5|110),
'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
);
# cleanup
-
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy");
$node_subscriber2->safe_psql('postgres',
"DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
$node_publisher->safe_psql('postgres',
"DROP PUBLICATION regress_pub_combo_nogen_to_gen");
#####################
-# TEST tab_order:
+# TEST tab_order replication
#
# publisher-side cols 'b' and 'c' are generated
# subscriber-side col 'b' is not generated and col 'c' is generated.
@@ -498,7 +534,7 @@ is( $result, qq(4|8|88
'replicate generated columns with different order on the subscriber');
#####################
-# TEST tab_alter
+# TEST tab_alter replication
#
# Add a new table to existing publication, then
# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
@@ -516,7 +552,7 @@ is( $result, qq(1||22
3||66), 'add new table to existing publication');
#####################
-# TEST tabl_alter
+# TEST tab_alter
#
# Drop the generated column's expression on subscriber side.
# This changes the generated column into a non-generated column.
@@ -547,7 +583,6 @@ $node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_misc");
#####################
# try it with a subscriber-side trigger
-
$node_subscriber->safe_psql(
'postgres', q{
CREATE FUNCTION tab1_trigger_func() RETURNS trigger
@@ -568,7 +603,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('regress_sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-05 03:44 Peter Smith <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-08-05 03:44 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi,
Writing many new test case combinations has exposed a possible bug in
patch 0001.
In my previous post [1] there was questionable behaviour when
replicating from a normal (not generated) column on the publisher side
to a generated column on the subscriber side. Initially, I thought the
test might have exposed a possible PG17 bug, but now I think it has
really found a bug in patch 0001.
~~~
Previously (PG17) this would fail consistently both during COPY and
during normal replication.Now, patch 0001 has changed this behaviour
-- it is not always failing anymore.
The patch should not be impacting this existing behaviour. It only
introduces a new 'include_generated_columns', but since the publisher
side is not a generated column I do not expect there should be any
difference in behaviour for this test case. IMO the TAP test expected
results should be corrected for this scenario. And fix the bug.
Below is an example demonstrating PG17 behaviour.
======
Publisher:
----------
(notice column "b" is not generated)
test_pub=# CREATE TABLE tab_nogen_to_gen (a int, b int);
CREATE TABLE
test_pub=# INSERT INTO tab_nogen_to_gen VALUES (1,101),(2,102);
INSERT 0 2
test_pub=# CREATE PUBLICATION pub1 for TABLE tab_nogen_to_gen;
CREATE PUBLICATION
test_pub=#
Subscriber:
-----------
(notice corresponding column "b" is generated)
test_sub=# CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED
ALWAYS AS (a * 22) STORED);
CREATE TABLE
test_sub=#
Try to create a subscription. Notice we get the error: ERROR: logical
replication target relation "public.tab_nogen_to_gen" is missing
replicated column: "b"
test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
PUBLICATION pub1;
2024-08-05 13:16:40.043 AEST [20957] WARNING: subscriptions created
by regression test cases should have names starting with "regress_"
WARNING: subscriptions created by regression test cases should have
names starting with "regress_"
NOTICE: created replication slot "sub1" on publisher
CREATE SUBSCRIPTION
test_sub=# 2024-08-05 13:16:40.105 AEST [29258] LOG: logical
replication apply worker for subscription "sub1" has started
2024-08-05 13:16:40.117 AEST [29260] LOG: logical replication table
synchronization worker for subscription "sub1", table
"tab_nogen_to_gen" has started
2024-08-05 13:16:40.172 AEST [29260] ERROR: logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:16:40.173 AEST [20039] LOG: background worker "logical
replication tablesync worker" (PID 29260) exited with exit code 1
2024-08-05 13:16:45.187 AEST [29400] LOG: logical replication table
synchronization worker for subscription "sub1", table
"tab_nogen_to_gen" has started
2024-08-05 13:16:45.285 AEST [29400] ERROR: logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:16:45.286 AEST [20039] LOG: background worker "logical
replication tablesync worker" (PID 29400) exited with exit code 1
...
Create the subscription again, but this time with copy_data = false
test_sub=# CREATE SUBSCRIPTION sub1_nocopy CONNECTION
'dbname=test_pub' PUBLICATION pub1 WITH (copy_data = false);
2024-08-05 13:22:57.719 AEST [20957] WARNING: subscriptions created
by regression test cases should have names starting with "regress_"
WARNING: subscriptions created by regression test cases should have
names starting with "regress_"
NOTICE: created replication slot "sub1_nocopy" on publisher
CREATE SUBSCRIPTION
test_sub=# 2024-08-05 13:22:57.765 AEST [7012] LOG: logical
replication apply worker for subscription "sub1_nocopy" has started
test_sub=#
~~~
Then insert data from the publisher to see what happens for normal replication.
test_pub=#
test_pub=# INSERT INTO tab_nogen_to_gen VALUES (3,103),(4,104);
INSERT 0 2
~~~
Notice the subscriber gets the same error as before: ERROR: logical
replication target relation "public.tab_nogen_to_gen" is missing
replicated column: "b"
2024-08-05 13:25:14.897 AEST [20039] LOG: background worker "logical
replication apply worker" (PID 10957) exited with exit code 1
2024-08-05 13:25:19.933 AEST [11095] LOG: logical replication apply
worker for subscription "sub1_nocopy" has started
2024-08-05 13:25:19.966 AEST [11095] ERROR: logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:25:19.966 AEST [11095] CONTEXT: processing remote data
for replication origin "pg_16390" during message type "INSERT" in
transaction 742, finished at 0/1967BB0
2024-08-05 13:25:19.968 AEST [20039] LOG: background worker "logical
replication apply worker" (PID 11095) exited with exit code 1
2024-08-05 13:25:24.917 AEST [11225] LOG: logical replication apply
worker for subscription "sub1_nocopy" has started
2024-08-05 13:25:24.926 AEST [11225] ERROR: logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:25:24.926 AEST [11225] CONTEXT: processing remote data
for replication origin "pg_16390" during message type "INSERT" in
transaction 742, finished at 0/1967BB0
2024-08-05 13:25:24.927 AEST [20039] LOG: background worker "logical
replication apply worker" (PID 11225) exited with exit code 1
...
======
[1] https://www.postgresql.org/message-id/CAHut%2BPvtT8fKOfvVYr4vANx_fr92vedas%2BZRbQxvMC097rks6w%40mail...
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-07 05:26 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Shubham Khanna @ 2024-08-07 05:26 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Aug 5, 2024 at 8:10 AM Peter Smith <[email protected]> wrote:
>
> Hi Shubhab.
>
> Here are some more review comments for the v23-0001.
>
> ======
> 011_generated.pl b/src/test/subscription/t/011_generated.pl
>
> nitpick - renamed /regress_pub/regress_pub_tab1/ and
> /regress_sub1/regress_sub1_tab1/
> nitpick - typo /inital data /initial data/
> nitpick - typo /snode_subscriber2/node_subscriber2/
> nitpick - tweak the combo initial sync comments and messages
> nitpick - /#Cleanup/# cleanup/
> nitpick - tweak all the combo normal replication comments
> nitpick - removed blank line at the end
>
> ~~~
>
> 1. Refactor tab_gen_to_missing initial sync tests.
>
> I moved the tab_gen_to_missing initial sync for node_subscriber2 to be
> back where all the other initial sync tests are done.
> See the nitpicks patch file.
>
> ~~~
>
> 2. Refactor tab_nogen_to_gen initial sync tests
>
> I moved all the tab_nogen_to_gen initial sync tests back to where the
> other initial sync tests are done.
> See the nitpicks patch file.
>
> ~~~
>
> 3. Added another test case:
>
> Because the (current PG17) nogen-to-gen initial sync test case (with
> copy_data=true) gives an ERROR, I have added another combination to
> cover normal replication (e.g. using copy_data=false).
> See the nitpicks patch file.
>
> (This has exposed an inconsistency which IMO might be a PG17 bug. I
> have included TAP test comments about this, and plan to post a
> separate thread for it later).
>
> ~
>
> 4. GUC
>
> Moving and adding more CREATE SUBSCRIPTION exceeded some default GUCs,
> so extra configuration was needed.
> See the nitpick patch file.
>
I have fixed all the comments. The attached Patch(v24-0001) contains
all the changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v24-0001-Enable-support-for-include_generated_columns-opt.patch (106.6K, ../../CAHv8RjLf4afJVTfo6N5sDoUOKVQN4NVKP5L3Wj6YiVN-v1gPkQ@mail.gmail.com/2-v24-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 4413fa5357d52bc6439733bc6b9eac63be4cf47f Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Wed, 24 Jul 2024 11:22:07 +0530
Subject: [PATCH v24 1/2] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr'
PUBLICATION regress_pub_combo WITH (include_generated_columns = true,
copy_data = false)
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +
contrib/test_decoding/test_decoding.c | 26 +-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++---
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 531 +++++++++++++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 905 insertions(+), 141 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 79cd599692..3320c25a60 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3322,6 +3322,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6540,8 +6551,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..819a124c63 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..5de1531567 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 6dc54c7283..f40f61ed7a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4376,6 +4376,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 79190470f7..106f313fb1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4800,6 +4800,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4872,11 +4873,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4915,6 +4922,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4961,6 +4969,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5207,6 +5217,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..28752ade7e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 024469474d..3c7e563807 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..13499a155d 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -12,12 +12,30 @@ use Test::More;
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf(
+ 'postgresql.conf',
+ "max_wal_senders = 20
+ max_replication_slots = 20");
$node_publisher->start;
+# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber->start;
+# All subscribers on this node will use parameter include_generated_columns = true
+my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$node_subscriber2->init;
+$node_subscriber2->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
+$node_subscriber2->start;
+
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
@@ -28,32 +46,255 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab_gen_to_gen:
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', with different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_gen_to_nogen:
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+
+# tab_gen_to_missing:
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+
+# tab_missing_to_gen:
+# publisher-side col 'b' is missing.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_nogen_to_gen:
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_order:
+# publisher-side has generated cols 'b' and 'c'.
+# subscriber-side has non-generated col 'b', and generated-col 'c'.
+# columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab_alter:
+# for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+
+# create publications
+#
+# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_tab1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_gen_to_missing FOR TABLE tab_gen_to_missing"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_nogen_to_gen FOR TABLE tab_nogen_to_gen"
+);
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+
+# create subscriptions
+#
+# Note that all subscriptions created on node_subscriber2 use copy_data = false,
+# because copy_data = true with include_generated_columns is not yet supported.
+# For this reason, the expected inital data on node_subscriber2 is always empty.
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_tab1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_tab1"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+# Note, regress_sub1_combo_nogen_to_gen is not created here due to expected errors. See later.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+);
+
+#####################
# Wait for initial sync of all subscriptions
+#####################
+
$node_subscriber->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+#####################
+# TEST tab_gen_to_gen initial sync
+#####################
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_gen_to_gen initial sync, when include_generated_columns=false');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is($result, qq(),
+ 'tab_gen_to_gen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_gen_to_nogen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'tab_gen_to_nogen, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is($result, qq(),
+ 'tab_gen_to_nogen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_gen_to_missing initial sync
+#####################
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+# Note, the following is expected to work only because copy_data = false
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing");
+is($result, qq(),
+ 'tab_gen_to_missing initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_missing_to_gen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
+);
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is($result, qq(),
+ 'tab_missing_to_gen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_nogen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? column "b" is a generated column/, $offset);
+# Note, the following is expected to work only because copy_data = false
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen");
+is($result, qq(),
+ 'tab_nogen_to_gen initial sync, when include_generated_columns=true');
+
+# tab_order:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is($result, qq(), 'generated column initial sync');
+
+# tab_alter:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is($result, qq(), 'unsubscribed table initial data');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -62,6 +303,288 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#####################
+# TEST tab_gen_to_gen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', using a different computation.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+#$node_publisher->wait_for_catchup('regress_pub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm generated columns are not replicated when include_generated_columns=false'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm generated columns are not replicated when the subscriber-side column is also generated'
+);
+
+#####################
+# TEST tab_gen_to_nogen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is replicated.
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#####################
+# TEST tab_gen_to_missing replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
+
+# regress_sub1_combo_gen_to_missing: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo_gen_to_missing');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5),
+ 'missing generated column, include_generated_columns = false');
+
+# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated and it will throw an error.
+my $offset2 = -s $node_subscriber2->logfile;
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_gen_to_missing");
+
+#####################
+# TEST tab_missing_to_gen replication
+#
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo");
+
+#####################
+# TEST tab_nogen_to_gen replication
+#
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+#####################
+
+# When copy_data=true a COPY error occurred. Try again but with copy_data=false.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false, copy_data = false)"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
+#
+# XXX
+# The test below shows that current PG17 behavior does not give an error,
+# But this conflicts with the copy_data=true behavior so it might be a PG17 bug.
+# Needs more study.
+$node_publisher->wait_for_catchup('regress_sub1_combo_nogen_to_gen_nocopy');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
+);
+
+# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
+#
+# XXX
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
+$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_nogen_to_gen");
+
+#####################
+# TEST tab_order replication
+#
+# publisher-side cols 'b' and 'c' are generated
+# subscriber-side col 'b' is not generated and col 'c' is generated.
+# But pub/sub table cols are in different order.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order VALUES (4), (5)");
+
+# regress_sub2_misc: (include_generated_columns = true)
+# Confirm that depsite different orders replication occurs to the correct columns
+$node_publisher->wait_for_catchup('regress_sub2_misc');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is( $result, qq(4|8|88
+5|10|110),
+ 'replicate generated columns with different order on the subscriber');
+
+#####################
+# TEST tab_alter replication
+#
+# Add a new table to existing publication, then
+# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#####################
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION regress_sub2_misc REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('regress_sub2_misc');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66), 'add new table to existing publication');
+
+#####################
+# TEST tab_alter
+#
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#####################
+
+# change a gencol to a nogen col
+$node_subscriber2->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN c DROP EXPRESSION");
+
+# insert some data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (4), (5)");
+
+# confirm that replication now works for the subscriber nogen col
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_misc");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_misc");
+
+#####################
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
@@ -84,7 +607,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-07 05:37 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-08-07 05:37 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Aug 5, 2024 at 9:15 AM Peter Smith <[email protected]> wrote:
>
> Hi,
>
> Writing many new test case combinations has exposed a possible bug in
> patch 0001.
>
> In my previous post [1] there was questionable behaviour when
> replicating from a normal (not generated) column on the publisher side
> to a generated column on the subscriber side. Initially, I thought the
> test might have exposed a possible PG17 bug, but now I think it has
> really found a bug in patch 0001.
>
> ~~~
>
> Previously (PG17) this would fail consistently both during COPY and
> during normal replication.Now, patch 0001 has changed this behaviour
> -- it is not always failing anymore.
>
> The patch should not be impacting this existing behaviour. It only
> introduces a new 'include_generated_columns', but since the publisher
> side is not a generated column I do not expect there should be any
> difference in behaviour for this test case. IMO the TAP test expected
> results should be corrected for this scenario. And fix the bug.
>
> Below is an example demonstrating PG17 behaviour.
>
> ======
>
>
> Publisher:
> ----------
>
> (notice column "b" is not generated)
>
> test_pub=# CREATE TABLE tab_nogen_to_gen (a int, b int);
> CREATE TABLE
> test_pub=# INSERT INTO tab_nogen_to_gen VALUES (1,101),(2,102);
> INSERT 0 2
> test_pub=# CREATE PUBLICATION pub1 for TABLE tab_nogen_to_gen;
> CREATE PUBLICATION
> test_pub=#
>
> Subscriber:
> -----------
>
> (notice corresponding column "b" is generated)
>
> test_sub=# CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED
> ALWAYS AS (a * 22) STORED);
> CREATE TABLE
> test_sub=#
>
> Try to create a subscription. Notice we get the error: ERROR: logical
> replication target relation "public.tab_nogen_to_gen" is missing
> replicated column: "b"
>
> test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
> PUBLICATION pub1;
> 2024-08-05 13:16:40.043 AEST [20957] WARNING: subscriptions created
> by regression test cases should have names starting with "regress_"
> WARNING: subscriptions created by regression test cases should have
> names starting with "regress_"
> NOTICE: created replication slot "sub1" on publisher
> CREATE SUBSCRIPTION
> test_sub=# 2024-08-05 13:16:40.105 AEST [29258] LOG: logical
> replication apply worker for subscription "sub1" has started
> 2024-08-05 13:16:40.117 AEST [29260] LOG: logical replication table
> synchronization worker for subscription "sub1", table
> "tab_nogen_to_gen" has started
> 2024-08-05 13:16:40.172 AEST [29260] ERROR: logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:16:40.173 AEST [20039] LOG: background worker "logical
> replication tablesync worker" (PID 29260) exited with exit code 1
> 2024-08-05 13:16:45.187 AEST [29400] LOG: logical replication table
> synchronization worker for subscription "sub1", table
> "tab_nogen_to_gen" has started
> 2024-08-05 13:16:45.285 AEST [29400] ERROR: logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:16:45.286 AEST [20039] LOG: background worker "logical
> replication tablesync worker" (PID 29400) exited with exit code 1
> ...
>
> Create the subscription again, but this time with copy_data = false
>
> test_sub=# CREATE SUBSCRIPTION sub1_nocopy CONNECTION
> 'dbname=test_pub' PUBLICATION pub1 WITH (copy_data = false);
> 2024-08-05 13:22:57.719 AEST [20957] WARNING: subscriptions created
> by regression test cases should have names starting with "regress_"
> WARNING: subscriptions created by regression test cases should have
> names starting with "regress_"
> NOTICE: created replication slot "sub1_nocopy" on publisher
> CREATE SUBSCRIPTION
> test_sub=# 2024-08-05 13:22:57.765 AEST [7012] LOG: logical
> replication apply worker for subscription "sub1_nocopy" has started
>
> test_sub=#
>
> ~~~
>
> Then insert data from the publisher to see what happens for normal replication.
>
> test_pub=#
> test_pub=# INSERT INTO tab_nogen_to_gen VALUES (3,103),(4,104);
> INSERT 0 2
>
> ~~~
>
> Notice the subscriber gets the same error as before: ERROR: logical
> replication target relation "public.tab_nogen_to_gen" is missing
> replicated column: "b"
>
> 2024-08-05 13:25:14.897 AEST [20039] LOG: background worker "logical
> replication apply worker" (PID 10957) exited with exit code 1
> 2024-08-05 13:25:19.933 AEST [11095] LOG: logical replication apply
> worker for subscription "sub1_nocopy" has started
> 2024-08-05 13:25:19.966 AEST [11095] ERROR: logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:25:19.966 AEST [11095] CONTEXT: processing remote data
> for replication origin "pg_16390" during message type "INSERT" in
> transaction 742, finished at 0/1967BB0
> 2024-08-05 13:25:19.968 AEST [20039] LOG: background worker "logical
> replication apply worker" (PID 11095) exited with exit code 1
> 2024-08-05 13:25:24.917 AEST [11225] LOG: logical replication apply
> worker for subscription "sub1_nocopy" has started
> 2024-08-05 13:25:24.926 AEST [11225] ERROR: logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:25:24.926 AEST [11225] CONTEXT: processing remote data
> for replication origin "pg_16390" during message type "INSERT" in
> transaction 742, finished at 0/1967BB0
> 2024-08-05 13:25:24.927 AEST [20039] LOG: background worker "logical
> replication apply worker" (PID 11225) exited with exit code 1
>
This is an expected behaviour. The error message here is improvised.
This error is consistent and it is being handled in the 0002 patch.
Below are the logs for the same:
2024-08-07 10:47:45.977 IST [29756] LOG: logical replication table
synchronization worker for subscription "sub1", table
"tab_nogen_to_gen" has started
2024-08-07 10:47:46.116 IST [29756] ERROR: logical replication target
relation "public.tab_nogen_to_gen" has a generated column "b" but
corresponding column on source relation is not a generated column
0002 Patch needs to be applied to get rid of this error.
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-07 08:00 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
0 siblings, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-08-07 08:00 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham,
Here are my review comments for patch v24-0001
I think the TAP tests have incorrect expected results for the nogen-to-gen case.
Whereas the HEAD code will cause "ERROR" for this test scenario, patch
0001 does not. IMO the behaviour should be unchanged for this scenario
which has no generated column on the publisher side. So it seems this
is a bug in patch 0001.
FYI, I have included "FIXME" comments in the attached top-up diff
patch to show which test cases I think are expecting wrong results.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 13499a1..a9430f7 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -478,10 +478,11 @@ $node_publisher->safe_psql('postgres',
# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
#
-# XXX
-# The test below shows that current PG17 behavior does not give an error,
-# But this conflicts with the copy_data=true behavior so it might be a PG17 bug.
-# Needs more study.
+# FIXME
+# I think the following expected result is wrong. IIUC it should give
+# the same error as HEAD -- e.g. something like:
+# ERROR: logical replication target relation "public.tab_nogen_to_gen" is missing
+# replicated column: "b"
$node_publisher->wait_for_catchup('regress_sub1_combo_nogen_to_gen_nocopy');
$result =
$node_subscriber->safe_psql('postgres',
@@ -495,9 +496,11 @@ is( $result, qq(4|88
# When copy_data=false, no COPY error occurs.
# The col 'b' is not replicated; the subscriber-side generated value is inserted.
#
-# XXX
-# It is correct for this to give the same result as above, but it needs more
-# study to determine if the above result was actually correct, or a PG17 bug.
+# FIXME
+# Since there is no generated column on the publishing side this should give
+# the same result as the previous test. -- e.g. something like:
+# ERROR: logical replication target relation "public.tab_nogen_to_gen" is missing
+# replicated column: "b"
$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres',
Attachments:
[text/plain] PS_NITPICKS_20240807_gencols_v240001.txt (1.6K, ../../CAHut+PvZWOYgUFCC2FFcbKoPUG65f8oz3bf90fw7CmejG60PKQ@mail.gmail.com/2-PS_NITPICKS_20240807_gencols_v240001.txt)
download | inline diff:
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 13499a1..a9430f7 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -478,10 +478,11 @@ $node_publisher->safe_psql('postgres',
# regress_sub1_combo_nogen_to_gen: (include_generated_columns = false)
#
-# XXX
-# The test below shows that current PG17 behavior does not give an error,
-# But this conflicts with the copy_data=true behavior so it might be a PG17 bug.
-# Needs more study.
+# FIXME
+# I think the following expected result is wrong. IIUC it should give
+# the same error as HEAD -- e.g. something like:
+# ERROR: logical replication target relation "public.tab_nogen_to_gen" is missing
+# replicated column: "b"
$node_publisher->wait_for_catchup('regress_sub1_combo_nogen_to_gen_nocopy');
$result =
$node_subscriber->safe_psql('postgres',
@@ -495,9 +496,11 @@ is( $result, qq(4|88
# When copy_data=false, no COPY error occurs.
# The col 'b' is not replicated; the subscriber-side generated value is inserted.
#
-# XXX
-# It is correct for this to give the same result as above, but it needs more
-# study to determine if the above result was actually correct, or a PG17 bug.
+# FIXME
+# Since there is no generated column on the publishing side this should give
+# the same result as the previous test. -- e.g. something like:
+# ERROR: logical replication target relation "public.tab_nogen_to_gen" is missing
+# replicated column: "b"
$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
$result =
$node_subscriber2->safe_psql('postgres',
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-08 05:23 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-08-08 05:23 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Aug 7, 2024 at 1:31 PM Peter Smith <[email protected]> wrote:
>
> Hi Shubham,
>
> Here are my review comments for patch v24-0001
>
> I think the TAP tests have incorrect expected results for the nogen-to-gen case.
>
> Whereas the HEAD code will cause "ERROR" for this test scenario, patch
> 0001 does not. IMO the behaviour should be unchanged for this scenario
> which has no generated column on the publisher side. So it seems this
> is a bug in patch 0001.
>
> FYI, I have included "FIXME" comments in the attached top-up diff
> patch to show which test cases I think are expecting wrong results.
>
Fixed all the comments. The attached Patch(v25-0001) contains all the changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v25-0001-Enable-support-for-include_generated_columns-opt.patch (106.3K, ../../CAHv8Rj+v-TJ3RoZse87zyVPr-VhrOXDw55-3F3xF1+6WSJxaMA@mail.gmail.com/2-v25-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 85807e4e8bda8980c187afe0a76e3869dddbea3a Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Wed, 24 Jul 2024 11:22:07 +0530
Subject: [PATCH v25] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr'
PUBLICATION regress_pub_combo WITH (include_generated_columns = true,
copy_data = false)
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +
contrib/test_decoding/test_decoding.c | 26 +-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/relation.c | 2 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++---
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 522 +++++++++++++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
30 files changed, 896 insertions(+), 141 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 79cd599692..3320c25a60 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3322,6 +3322,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6540,8 +6551,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 0602398a54..f611148472 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -506,7 +506,6 @@ publication_translate_columns(Relation targetrel, List *columns,
Bitmapset *set = NULL;
ListCell *lc;
int n = 0;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
/* Bail out when no column list defined. */
if (!columns)
@@ -534,12 +533,6 @@ publication_translate_columns(Relation targetrel, List *columns,
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1232,7 +1225,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..819a124c63 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index f139e7b01e..7387850db9 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
int attnum;
Form_pg_attribute attr = TupleDescAttr(desc, i);
- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped || (attr->attgenerated && !MySubscription->includegencols))
{
entry->attrmap->attnums[i] = -1;
continue;
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 6dc54c7283..f40f61ed7a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4376,6 +4376,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b6e01d3d29..db5dd66c11 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4847,6 +4847,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4919,11 +4920,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4962,6 +4969,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -5008,6 +5016,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5254,6 +5264,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..28752ade7e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 024469474d..3c7e563807 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 30b6371134..aa1450315d 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 479d4f3264..b1899ddb1a 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..eab0d9b541 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -12,12 +12,30 @@ use Test::More;
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf(
+ 'postgresql.conf',
+ "max_wal_senders = 20
+ max_replication_slots = 20");
$node_publisher->start;
+# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber->start;
+# All subscribers on this node will use parameter include_generated_columns = true
+my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$node_subscriber2->init;
+$node_subscriber2->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
+$node_subscriber2->start;
+
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
@@ -28,32 +46,256 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab_gen_to_gen:
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', with different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_gen_to_nogen:
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+
+# tab_gen_to_missing:
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+
+# tab_missing_to_gen:
+# publisher-side col 'b' is missing.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_nogen_to_gen:
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_order:
+# publisher-side has generated cols 'b' and 'c'.
+# subscriber-side has non-generated col 'b', and generated-col 'c'.
+# columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab_alter:
+# for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+
+# create publications
+#
+# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_tab1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_gen, tab_gen_to_nogen, tab_missing_to_gen"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_gen_to_missing FOR TABLE tab_gen_to_missing"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_nogen_to_gen FOR TABLE tab_nogen_to_gen"
+);
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+
+# create subscriptions
+#
+# Note that all subscriptions created on node_subscriber2 use copy_data = false,
+# because copy_data = true with include_generated_columns is not yet supported.
+# For this reason, the expected inital data on node_subscriber2 is always empty.
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_tab1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_tab1"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+# Note, regress_sub1_combo_nogen_to_gen is not created here due to expected errors. See later.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+);
+
+#####################
# Wait for initial sync of all subscriptions
+#####################
+
$node_subscriber->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+#####################
+# TEST tab_gen_to_gen initial sync
+#####################
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_gen_to_gen initial sync, when include_generated_columns=false');
+$result =
+ $node_subscriber2->safe_psql('postgres', "SELECT a, b FROM tab_gen_to_gen");
+is($result, qq(),
+ 'tab_gen_to_gen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_gen_to_nogen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'tab_gen_to_nogen, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is($result, qq(),
+ 'tab_gen_to_nogen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_gen_to_missing initial sync
+#####################
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+# Note, the following is expected to work only because copy_data = false
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing");
+is($result, qq(),
+ 'tab_gen_to_missing initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_missing_to_gen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
+);
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is($result, qq(),
+ 'tab_missing_to_gen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_nogen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false)"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+# Note, the following is expected to work only because copy_data = false
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen");
+is($result, qq(),
+ 'tab_nogen_to_gen initial sync, when include_generated_columns=true');
+
+# tab_order:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is($result, qq(), 'generated column initial sync');
+
+# tab_alter:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is($result, qq(), 'unsubscribed table initial data');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -62,6 +304,278 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#####################
+# TEST tab_gen_to_gen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', using a different computation.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+#$node_publisher->wait_for_catchup('regress_pub_combo');
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm generated columns are not replicated when include_generated_columns=false'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated. We can know this because the result
+# value is the subscriber-side computation (which is different from the
+# publisher-side computation for this column).
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm generated columns are not replicated when the subscriber-side column is also generated'
+);
+
+#####################
+# TEST tab_gen_to_nogen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is replicated.
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#####################
+# TEST tab_gen_to_missing replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
+
+# regress_sub1_combo_gen_to_missing: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo_gen_to_missing');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5),
+ 'missing generated column, include_generated_columns = false');
+
+# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated and it will throw an error.
+my $offset2 = -s $node_subscriber2->logfile;
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_gen_to_missing");
+
+#####################
+# TEST tab_missing_to_gen replication
+#
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo");
+
+#####################
+# TEST tab_nogen_to_gen replication
+#
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+#####################
+
+# When copy_data=true a COPY error occurred. Try again but with copy_data=false.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = false, copy_data = false)"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+
+
+# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
+#
+# XXX
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
+$node_publisher->wait_for_catchup('regress_sub2_combo_nogen_to_gen');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_nogen_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is not generated, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen_nocopy");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_nogen_to_gen");
+
+#####################
+# TEST tab_order replication
+#
+# publisher-side cols 'b' and 'c' are generated
+# subscriber-side col 'b' is not generated and col 'c' is generated.
+# But pub/sub table cols are in different order.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order VALUES (4), (5)");
+
+# regress_sub2_misc: (include_generated_columns = true)
+# Confirm that depsite different orders replication occurs to the correct columns
+$node_publisher->wait_for_catchup('regress_sub2_misc');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is( $result, qq(4|8|88
+5|10|110),
+ 'replicate generated columns with different order on the subscriber');
+
+#####################
+# TEST tab_alter replication
+#
+# Add a new table to existing publication, then
+# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#####################
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION regress_sub2_misc REFRESH PUBLICATION");
+$node_publisher->wait_for_catchup('regress_sub2_misc');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66), 'add new table to existing publication');
+
+#####################
+# TEST tab_alter
+#
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#####################
+
+# change a gencol to a nogen col
+$node_subscriber2->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN c DROP EXPRESSION");
+
+# insert some data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (4), (5)");
+
+# confirm that replication now works for the subscriber nogen col
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is( $result, qq(1||22
+2||44
+3||66
+4|8|8
+5|10|10), 'after drop generated column expression');
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_misc");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_misc");
+
+#####################
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
@@ -84,7 +598,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.34.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-08 07:13 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-08-08 07:13 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham,
I think the v25-0001 patch only half-fixes the problems reported in my
v24-0001 review.
~
Background (from the commit message):
This commit enables support for the 'include_generated_columns' option
in logical replication, allowing the transmission of generated column
information and data alongside regular table changes.
~
The broken TAP test scenario in question is replicating from a
"not-generated" column to a "generated" column. As the generated
column is not on the publishing side, IMO the
'include_generated_columns' option should have zero effect here.
In other words, I expect this TAP test for 'include_generated_columns
= true' case should also be failing, as I wrote already yesterday:
+# FIXME
+# Since there is no generated column on the publishing side this should give
+# the same result as the previous test. -- e.g. something like:
+# ERROR: logical replication target relation
"public.tab_nogen_to_gen" is missing
+# replicated column: "b"
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-10 14:23 vignesh C <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: vignesh C @ 2024-08-10 14:23 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Peter Smith <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, 8 Aug 2024 at 10:53, Shubham Khanna <[email protected]> wrote:
>
> On Wed, Aug 7, 2024 at 1:31 PM Peter Smith <[email protected]> wrote:
> >
> > Hi Shubham,
> >
> > Here are my review comments for patch v24-0001
> >
> > I think the TAP tests have incorrect expected results for the nogen-to-gen case.
> >
> > Whereas the HEAD code will cause "ERROR" for this test scenario, patch
> > 0001 does not. IMO the behaviour should be unchanged for this scenario
> > which has no generated column on the publisher side. So it seems this
> > is a bug in patch 0001.
> >
> > FYI, I have included "FIXME" comments in the attached top-up diff
> > patch to show which test cases I think are expecting wrong results.
> >
>
> Fixed all the comments. The attached Patch(v25-0001) contains all the changes.
Few comments:
1) Can we add one test with replica identity full to show that
generated column is included in case of update operation with
test_decoding.
2) At the end of the file generated_columns.sql a newline is missing:
+-- when 'include-generated-columns' = '0' the generated column 'b'
values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
3)
3.a)This can be changed:
+-- when 'include-generated-columns' is not set the generated column
'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
to:
-- By default, 'include-generated-columns' is enabled, so the values
for the generated column 'b' will be replicated even if it is not
explicitly specified.
3.b) This can be changed:
-- when 'include-generated-columns' = '1' the generated column 'b'
values will be replicated
to:
-- when 'include-generated-columns' is enabled, the values of the
generated column 'b' will be replicated.
3.c) This can be changed:
-- when 'include-generated-columns' = '0' the generated column 'b'
values will not be replicated
to:
-- when 'include-generated-columns' is disabled, the values of the
generated column 'b' will not be replicated.
4) I did not see any test for dump, can we add one test for this.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-15 07:39 Peter Smith <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Peter Smith @ 2024-08-15 07:39 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; +Cc: Shubham Khanna <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Tue, Jul 23, 2024 at 9:23 AM Peter Smith <[email protected]> wrote:
>
> On Fri, Jul 19, 2024 at 4:01 PM Shlok Kyal <[email protected]> wrote:
> >
> > On Thu, 18 Jul 2024 at 13:55, Peter Smith <[email protected]> wrote:
> > >
> > > Hi, here are some review comments for v19-0002
> > > ======
> > > src/test/subscription/t/004_sync.pl
> > >
> > > 1.
> > > This new test is not related to generated columns. IIRC, this is just
> > > some test that we discovered missing during review of this thread. As
> > > such, I think this change can be posted/patched separately from this
> > > thread.
> > >
> > I have removed the test for this thread.
> >
> > I have also addressed the remaining comments for v19-0002 patch.
>
> Hi, I have no more review comments for patch v20-0002 at this time.
>
> I saw that the above test was removed from this thread as suggested,
> but I could not find that any new thread was started to propose this
> valuable missing test.
>
I still did not find any new thread for adding the missing test case,
so I started one myself [1].
======
[1] https://www.postgresql.org/message-id/[email protected]...
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-16 04:34 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-08-16 04:34 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <[email protected]> wrote:
>
> Hi Shubham,
>
> I think the v25-0001 patch only half-fixes the problems reported in my
> v24-0001 review.
>
> ~
>
> Background (from the commit message):
> This commit enables support for the 'include_generated_columns' option
> in logical replication, allowing the transmission of generated column
> information and data alongside regular table changes.
>
> ~
>
> The broken TAP test scenario in question is replicating from a
> "not-generated" column to a "generated" column. As the generated
> column is not on the publishing side, IMO the
> 'include_generated_columns' option should have zero effect here.
>
> In other words, I expect this TAP test for 'include_generated_columns
> = true' case should also be failing, as I wrote already yesterday:
>
> +# FIXME
> +# Since there is no generated column on the publishing side this should give
> +# the same result as the previous test. -- e.g. something like:
> +# ERROR: logical replication target relation
> "public.tab_nogen_to_gen" is missing
> +# replicated column: "b"
I have fixed the given comments. The attached v26-0001 Patch contains
the required changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v26-0001-Enable-support-for-include_generated_columns-opt.patch (106.0K, ../../CAHv8RjKH8MJhodtN5tXUD8pp=PKOomrnSUXcPCuwseu=bS75-Q@mail.gmail.com/2-v26-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From fba46a0eaf9a2a0595841062d70e01048958ec68 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Fri, 16 Aug 2024 02:20:56 +0530
Subject: [PATCH v26] Enable support for 'include_generated_columns' option
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr'
PUBLICATION regress_pub_combo WITH (include_generated_columns = true,
copy_data = false)
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 52 ++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 22 +
contrib/test_decoding/test_decoding.c | 26 +-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++---
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/011_generated.pl | 521 +++++++++++++++++-
src/test/subscription/t/031_column_list.pl | 6 +-
29 files changed, 894 insertions(+), 140 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..f3b26aa9e1
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,52 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+--------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4 b[integer]:8
+ table public.gencoltable: INSERT: a[integer]:5 b[integer]:10
+ table public.gencoltable: INSERT: a[integer]:6 b[integer]:12
+ COMMIT
+(5 rows)
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:7
+ table public.gencoltable: INSERT: a[integer]:8
+ table public.gencoltable: INSERT: a[integer]:9
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..6d6d1d6564
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,22 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- when 'include-generated-columns' is not set the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' = '1' the generated column 'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' = '0' the generated column 'b' values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 79cd599692..3320c25a60 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3322,6 +3322,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6540,8 +6551,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 7fe5fe2b86..00a66c12ce 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -511,7 +511,6 @@ pub_collist_validate(Relation targetrel, List *columns)
{
Bitmapset *set = NULL;
ListCell *lc;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
foreach(lc, columns)
{
@@ -530,12 +529,6 @@ pub_collist_validate(Relation targetrel, List *columns)
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1214,7 +1207,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b925c464ae..27c4d43ec4 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 245e9be6f2..6c145dc378 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4399,6 +4399,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b6e01d3d29..db5dd66c11 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4847,6 +4847,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4919,11 +4920,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4962,6 +4969,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -5008,6 +5016,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5254,6 +5264,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..28752ade7e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 024469474d..3c7e563807 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 660245ed0c..11f3fcc8f9 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index f68a5b5986..f344eafca3 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl1 (id, ctid);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..5f4b1e4cce 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -12,12 +12,30 @@ use Test::More;
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf(
+ 'postgresql.conf',
+ "max_wal_senders = 20
+ max_replication_slots = 20");
$node_publisher->start;
+# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber->start;
+# All subscribers on this node will use parameter include_generated_columns = true
+my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$node_subscriber2->init;
+$node_subscriber2->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
+$node_subscriber2->start;
+
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
@@ -28,32 +46,272 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab_gen_to_gen:
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', with different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_gen_to_nogen:
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+
+# tab_gen_to_missing:
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+
+# tab_missing_to_gen:
+# publisher-side col 'b' is missing.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_nogen_to_gen:
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_order:
+# publisher-side has generated cols 'b' and 'c'.
+# subscriber-side has non-generated col 'b', and generated-col 'c'.
+# columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab_alter:
+# for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+
+# create publications
+#
+# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_tab1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_nogen, tab_missing_to_gen"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_gen_to_missing FOR TABLE tab_gen_to_missing"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_nogen_to_gen FOR TABLE tab_nogen_to_gen"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_gen_to_gen FOR TABLE tab_gen_to_gen"
+);
+
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+
+# create subscriptions
+#
+# Note that all subscriptions created on node_subscriber2 use copy_data = false,
+# because copy_data = true with include_generated_columns is not yet supported.
+# For this reason, the expected inital data on node_subscriber2 is always empty.
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_tab1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_tab1"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+# Note, regress_sub1_combo_nogen_to_gen is not created here due to expected errors. See later.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
+);
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+);
+
+#####################
# Wait for initial sync of all subscriptions
+#####################
+
$node_subscriber->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+#####################
+# TEST tab_gen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_gen");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_gen");
+
+#####################
+# TEST tab_gen_to_nogen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'tab_gen_to_nogen, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is($result, qq(),
+ 'tab_gen_to_nogen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_gen_to_missing initial sync
+#####################
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+# Note, the following is expected to work only because copy_data = false
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing");
+is($result, qq(),
+ 'tab_gen_to_missing initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_missing_to_gen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
+);
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is($result, qq(),
+ 'tab_missing_to_gen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_nogen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+my $offset2 = -s $node_subscriber2->logfile;
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+
+# tab_order:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is($result, qq(), 'generated column initial sync');
+
+# tab_alter:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is($result, qq(), 'unsubscribed table initial data');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -62,6 +320,261 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#####################
+# TEST tab_gen_to_gen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', using a different computation.
+#####################
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_gen");
+
+# regress_sub2_combo_gen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
+#
+# XXX
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_gen_to_gen");
+
+#####################
+# TEST tab_gen_to_nogen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is replicated.
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#####################
+# TEST tab_gen_to_missing replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
+
+# regress_sub1_combo_gen_to_missing: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo_gen_to_missing');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5),
+ 'missing generated column, include_generated_columns = false');
+
+# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated and it will throw an error.
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_gen_to_missing");
+
+#####################
+# TEST tab_missing_to_gen replication
+#
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo");
+
+#####################
+# TEST tab_nogen_to_gen replication
+#
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+#####################
+
+# When copy_data=true a COPY error occurred. Try again but with copy_data=false.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen");
+
+# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
+#
+# XXX
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_nogen_to_gen");
+
+#####################
+# TEST tab_order replication
+#
+# publisher-side cols 'b' and 'c' are generated
+# subscriber-side col 'b' is not generated and col 'c' is generated.
+# But pub/sub table cols are in different order.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order VALUES (4), (5)");
+
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_order" is missing replicated column: "c"/,
+ $offset2);
+
+#####################
+# TEST tab_alter replication
+#
+# Add a new table to existing publication, then
+# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#####################
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION regress_sub2_misc REFRESH PUBLICATION");
+
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_order" is missing replicated column: "c"/,
+ $offset2);
+
+#####################
+# TEST tab_alter
+#
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#####################
+
+# change a gencol to a nogen col
+$node_subscriber2->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN c DROP EXPRESSION");
+
+# insert some data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (4), (5)");
+
+# confirm that replication now works for the subscriber nogen col
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is($result, qq(), 'after drop generated column expression');
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_misc");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_misc");
+
+#####################
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
@@ -84,7 +597,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-16 09:17 vignesh C <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: vignesh C @ 2024-08-16 09:17 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Peter Smith <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, 16 Aug 2024 at 10:04, Shubham Khanna
<[email protected]> wrote:
>
> On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <[email protected]> wrote:
> >
> > Hi Shubham,
> >
> > I think the v25-0001 patch only half-fixes the problems reported in my
> > v24-0001 review.
> >
> > ~
> >
> > Background (from the commit message):
> > This commit enables support for the 'include_generated_columns' option
> > in logical replication, allowing the transmission of generated column
> > information and data alongside regular table changes.
> >
> > ~
> >
> > The broken TAP test scenario in question is replicating from a
> > "not-generated" column to a "generated" column. As the generated
> > column is not on the publishing side, IMO the
> > 'include_generated_columns' option should have zero effect here.
> >
> > In other words, I expect this TAP test for 'include_generated_columns
> > = true' case should also be failing, as I wrote already yesterday:
> >
> > +# FIXME
> > +# Since there is no generated column on the publishing side this should give
> > +# the same result as the previous test. -- e.g. something like:
> > +# ERROR: logical replication target relation
> > "public.tab_nogen_to_gen" is missing
> > +# replicated column: "b"
>
> I have fixed the given comments. The attached v26-0001 Patch contains
> the required changes.
Few comments:
1) There's no need to pass include_generated_columns in this case; we
can retrieve it from ctx->data instead:
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns,
bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation,
TransactionId xid,
2) Commit message:
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
An error will occur in this case, so the message should be updated accordingly.
3) The current test is structured as follows: a) Create all required
tables b) Insert data into tables c) Create publications d) Create
subscriptions e) Perform inserts and verify
This approach can make reviewing and maintenance somewhat challenging.
Instead, could you modify it to: a) Create the required table for a
single test b) Insert data for this test c) Create the publication for
this test d) Create the subscriptions for this test e) Perform inserts
and verify f) Clean up
4) We can maintain the test as a separate 0002 patch, as it may need a
few rounds of review and final adjustments. Once it's fully completed,
we can merge it back in.
5) Once we create and drop publication/subscriptions for individual
tests, we won't need such extensive configuration; we should be able
to run them with default values:
+$node_publisher->append_conf(
+ 'postgresql.conf',
+ "max_wal_senders = 20
+ max_replication_slots = 20");
Regards,
Vignesh
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-16 11:10 Shubham Khanna <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-08-16 11:10 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Peter Smith <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Sat, Aug 10, 2024 at 7:53 PM vignesh C <[email protected]> wrote:
>
> On Thu, 8 Aug 2024 at 10:53, Shubham Khanna <[email protected]> wrote:
> >
> > On Wed, Aug 7, 2024 at 1:31 PM Peter Smith <[email protected]> wrote:
> > >
> > > Hi Shubham,
> > >
> > > Here are my review comments for patch v24-0001
> > >
> > > I think the TAP tests have incorrect expected results for the nogen-to-gen case.
> > >
> > > Whereas the HEAD code will cause "ERROR" for this test scenario, patch
> > > 0001 does not. IMO the behaviour should be unchanged for this scenario
> > > which has no generated column on the publisher side. So it seems this
> > > is a bug in patch 0001.
> > >
> > > FYI, I have included "FIXME" comments in the attached top-up diff
> > > patch to show which test cases I think are expecting wrong results.
> > >
> >
> > Fixed all the comments. The attached Patch(v25-0001) contains all the changes.
>
> Few comments:
> 1) Can we add one test with replica identity full to show that
> generated column is included in case of update operation with
> test_decoding.
>
> 2) At the end of the file generated_columns.sql a newline is missing:
> +-- when 'include-generated-columns' = '0' the generated column 'b'
> values will not be replicated
> +INSERT INTO gencoltable (a) VALUES (7), (8), (9);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '0');
> +
> +DROP TABLE gencoltable;
> +
> +SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
> \ No newline at end of file
>
> 3)
> 3.a)This can be changed:
> +-- when 'include-generated-columns' is not set the generated column
> 'b' values will be replicated
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
>
> to:
> -- By default, 'include-generated-columns' is enabled, so the values
> for the generated column 'b' will be replicated even if it is not
> explicitly specified.
>
> 3.b) This can be changed:
> -- when 'include-generated-columns' = '1' the generated column 'b'
> values will be replicated
> to:
> -- when 'include-generated-columns' is enabled, the values of the
> generated column 'b' will be replicated.
>
> 3.c) This can be changed:
> -- when 'include-generated-columns' = '0' the generated column 'b'
> values will not be replicated
> to:
> -- when 'include-generated-columns' is disabled, the values of the
> generated column 'b' will not be replicated.
>
> 4) I did not see any test for dump, can we add one test for this.
Fixed all the given comments. The attached Patch v27-0001 contains all
the required changes. Also, I have created a separate Patch (v27-0002)
for TAP Tests related to 'include-generated-columns'
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v27-0001-Enable-support-for-include_generated_columns-opt.patch (86.4K, ../../CAHv8RjLbH3DWZsxkbVZ0qsfc6QCf1Rh_Jgzz0sWsENE75GTg-g@mail.gmail.com/2-v27-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From e113a8e53bdfed1b6ef2e17e215e34796cfd8f20 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Fri, 16 Aug 2024 16:02:26 +0530
Subject: [PATCH v27 1/2] Enable support for 'include_generated_columns'
option`
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr'
PUBLICATION regress_pub_combo WITH (include_generated_columns = true,
copy_data = false)
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 59 +++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 27 +++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 41 +++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 11 ++
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/031_column_list.pl | 6 +-
29 files changed, 400 insertions(+), 136 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..48f900f069
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,59 @@
+-- test decoding of generated columns
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- By default, 'include-generated-columns' is enabled, so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+INSERT INTO gencoltable (a) VALUES (1);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ COMMIT
+(3 rows)
+
+-- when 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+INSERT INTO gencoltable (a) VALUES (2);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ COMMIT
+(3 rows)
+
+-- when 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+INSERT INTO gencoltable (a) VALUES (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:3
+ COMMIT
+(3 rows)
+
+-- with REPLICA IDENTITY = FULL, to show old-key data includes generated columns data for updates.
+ALTER TABLE gencoltable REPLICA IDENTITY FULL;
+UPDATE gencoltable SET a = 10;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: UPDATE: old-key: a[integer]:1 b[integer]:2 new-tuple: a[integer]:10 b[integer]:20
+ table public.gencoltable: UPDATE: old-key: a[integer]:2 b[integer]:4 new-tuple: a[integer]:10 b[integer]:20
+ table public.gencoltable: UPDATE: old-key: a[integer]:3 b[integer]:6 new-tuple: a[integer]:10 b[integer]:20
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..fb156c215f
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,27 @@
+-- test decoding of generated columns
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- column b' is a generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- By default, 'include-generated-columns' is enabled, so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+INSERT INTO gencoltable (a) VALUES (1);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- when 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+INSERT INTO gencoltable (a) VALUES (2);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- when 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+INSERT INTO gencoltable (a) VALUES (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+-- with REPLICA IDENTITY = FULL, to show old-key data includes generated columns data for updates.
+ALTER TABLE gencoltable REPLICA IDENTITY FULL;
+UPDATE gencoltable SET a = 10;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 79cd599692..3320c25a60 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3322,6 +3322,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6540,8 +6551,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 7fe5fe2b86..00a66c12ce 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -511,7 +511,6 @@ pub_collist_validate(Relation targetrel, List *columns)
{
Bitmapset *set = NULL;
ListCell *lc;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
foreach(lc, columns)
{
@@ -530,12 +529,6 @@ pub_collist_validate(Relation targetrel, List *columns)
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1214,7 +1207,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b925c464ae..27c4d43ec4 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 245e9be6f2..6c145dc378 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4399,6 +4399,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..4624649cd7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -86,7 +86,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -283,11 +284,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +399,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -731,11 +744,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns,
+ data->include_generated_columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx, relentry->columns);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns,
+ data->include_generated_columns);
if (data->in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +800,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1103,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1549,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b6e01d3d29..db5dd66c11 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4847,6 +4847,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4919,11 +4920,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4962,6 +4969,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -5008,6 +5016,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5254,6 +5264,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..28752ade7e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 5bcc2244d5..dde93d0406 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2983,6 +2983,17 @@ my %tests = (
like => { %full_runs, section_post_data => 1, },
},
+ 'CREATE SUBSCRIPTION sub4' => {
+ create_order => 50,
+ create_sql => 'CREATE SUBSCRIPTION sub4
+ CONNECTION \'dbname=postgres\' PUBLICATION pub1
+ WITH (connect = false, origin = any, include_generated_columns = true);',
+ regexp => qr/^
+ \QCREATE SUBSCRIPTION sub4 CONNECTION 'dbname=postgres' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub4', include_generated_columns = true);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ },
+
'ALTER PUBLICATION pub1 ADD TABLE test_table' => {
create_order => 51,
create_sql =>
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 024469474d..3c7e563807 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 660245ed0c..11f3fcc8f9 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index f68a5b5986..f344eafca3 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl1 (id, ctid);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
[application/octet-stream] v27-0002-Tap-tests-for-include-generated-columns.patch (21.9K, ../../CAHv8RjLbH3DWZsxkbVZ0qsfc6QCf1Rh_Jgzz0sWsENE75GTg-g@mail.gmail.com/3-v27-0002-Tap-tests-for-include-generated-columns.patch)
download | inline diff:
From ae61cda76dbc5d3666551a23607df2ea32821e54 Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Fri, 16 Aug 2024 16:04:30 +0530
Subject: [PATCH v27 2/2] Tap tests for 'include-generated-columns'
Tap tests for 'include-generated-columns'
---
src/test/subscription/t/011_generated.pl | 521 ++++++++++++++++++++++-
1 file changed, 517 insertions(+), 4 deletions(-)
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..5f4b1e4cce 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -12,12 +12,30 @@ use Test::More;
my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf(
+ 'postgresql.conf',
+ "max_wal_senders = 20
+ max_replication_slots = 20");
$node_publisher->start;
+# All subscribers on this node will use parameter include_generated_columns = false
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
$node_subscriber->start;
+# All subscribers on this node will use parameter include_generated_columns = true
+my $node_subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$node_subscriber2->init;
+$node_subscriber2->append_conf(
+ 'postgresql.conf',
+ "max_logical_replication_workers = 20
+ max_worker_processes = 20");
+$node_subscriber2->start;
+
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
@@ -28,32 +46,272 @@ $node_subscriber->safe_psql('postgres',
"CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
);
+# tab_gen_to_gen:
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', with different computation.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_gen_to_nogen:
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_nogen (a int, b int)");
+
+# tab_gen_to_missing:
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_gen_to_missing (a int)");
+
+# tab_missing_to_gen:
+# publisher-side col 'b' is missing.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_nogen_to_gen:
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
+# tab_order:
+# publisher-side has generated cols 'b' and 'c'.
+# subscriber-side has non-generated col 'b', and generated-col 'c'.
+# columns on publisher/subscriber are in a different order
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_order (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED, a int, b int)"
+);
+
+# tab_alter:
+# for testing ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a * 22) STORED)"
+);
+
# data for initial sync
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 (a) VALUES (1), (2), (3)");
$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (1), (2), (3)");
+
+# create publications
+#
+# pub_combo_gen_to_missing is not included in pub_combo, because some tests give errors.
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_tab1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo FOR TABLE tab_gen_to_nogen, tab_missing_to_gen"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_gen_to_missing FOR TABLE tab_gen_to_missing"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_nogen_to_gen FOR TABLE tab_nogen_to_gen"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_combo_gen_to_gen FOR TABLE tab_gen_to_gen"
+);
+
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub_misc FOR TABLE tab_order");
+
+# create subscriptions
+#
+# Note that all subscriptions created on node_subscriber2 use copy_data = false,
+# because copy_data = true with include_generated_columns is not yet supported.
+# For this reason, the expected inital data on node_subscriber2 is always empty.
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_tab1 CONNECTION '$publisher_connstr' PUBLICATION regress_pub_tab1"
+);
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo"
+);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing"
);
+# Note, regress_sub1_combo_nogen_to_gen is not created here due to expected errors. See later.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_missing CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_missing with (include_generated_columns = true, copy_data = false)"
+);
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_misc CONNECTION '$publisher_connstr' PUBLICATION regress_pub_misc WITH (include_generated_columns = true, copy_data = false)"
+);
+
+#####################
# Wait for initial sync of all subscriptions
+#####################
+
$node_subscriber->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
+#####################
+# TEST tab_gen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_gen");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_gen");
+
+#####################
+# TEST tab_gen_to_nogen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'tab_gen_to_nogen, when include_generated_columns=false');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is($result, qq(),
+ 'tab_gen_to_nogen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_gen_to_missing initial sync
+#####################
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+# Note, the following is expected to work only because copy_data = false
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing");
+is($result, qq(),
+ 'tab_gen_to_missing initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_missing_to_gen initial sync
+#####################
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
+);
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is($result, qq(),
+ 'tab_missing_to_gen initial sync, when include_generated_columns=true');
+
+#####################
+# TEST tab_nogen_to_gen initial sync
+#####################
+# The subscription is created here, because it causes the tablesync worker to restart repetitively.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen"
+);
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+my $offset2 = -s $node_subscriber2->logfile;
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+
+# tab_order:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_order ORDER BY a");
+is($result, qq(), 'generated column initial sync');
+
+# tab_alter:
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is($result, qq(), 'unsubscribed table initial data');
+
# data to replicate
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab1");
is( $result, qq(1|22|
@@ -62,6 +320,261 @@ is( $result, qq(1|22|
4|88|
6|132|), 'generated columns replicated');
+#####################
+# TEST tab_gen_to_gen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has generated col 'b', using a different computation.
+#####################
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_gen");
+
+# regress_sub2_combo_gen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
+#
+# XXX
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_gen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_gen_to_gen");
+
+#####################
+# TEST tab_gen_to_nogen replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side has non-generated col 'b'.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is replicated.
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+#####################
+# TEST tab_gen_to_missing replication
+#
+# publisher-side has generated col 'b'.
+# subscriber-side col 'b' is missing.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
+
+# regress_sub1_combo_gen_to_missing: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_combo_gen_to_missing');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5),
+ 'missing generated column, include_generated_columns = false');
+
+# regress_sub2_combo_gen_to_missing: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated and it will throw an error.
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_gen_to_missing");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_gen_to_missing");
+
+#####################
+# TEST tab_missing_to_gen replication
+#
+# publisher-side col 'b' is missing.
+# subscriber-side col 'b' is generated.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# regress_sub1_combo: (include_generated_columns = false)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub1_combo');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# regress_sub2_combo: (include_generated_columns = true)
+# Confirm that col 'b' is not replicated, but is generated as normal
+$node_publisher->wait_for_catchup('regress_sub2_combo');
+$result =
+ $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo");
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_combo");
+
+#####################
+# TEST tab_nogen_to_gen replication
+#
+# publisher-side has non-generated col 'b'.
+# subscriber-side has generated col 'b'.
+#####################
+
+# When copy_data=true a COPY error occurred. Try again but with copy_data=false.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub1_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen"
+);
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_combo_nogen_to_gen");
+
+# regress_sub2_combo_nogen_to_gen: (include_generated_columns = true)
+# When copy_data=false, no COPY error occurs.
+# The col 'b' is not replicated; the subscriber-side generated value is inserted.
+#
+# XXX
+# It is correct for this to give the same result as above, but it needs more
+# study to determine if the above result was actually correct, or a PG17 bug.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION regress_sub2_combo_nogen_to_gen CONNECTION '$publisher_connstr' PUBLICATION regress_pub_combo_nogen_to_gen WITH (include_generated_columns = true, copy_data = false)"
+);
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset2);
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_combo_nogen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_combo_nogen_to_gen");
+
+#####################
+# TEST tab_order replication
+#
+# publisher-side cols 'b' and 'c' are generated
+# subscriber-side col 'b' is not generated and col 'c' is generated.
+# But pub/sub table cols are in different order.
+#####################
+
+# insert data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_order VALUES (4), (5)");
+
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_order" is missing replicated column: "c"/,
+ $offset2);
+
+#####################
+# TEST tab_alter replication
+#
+# Add a new table to existing publication, then
+# do ALTER SUBSCRIPTION ... REFRESH PUBLICATION
+#####################
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION regress_pub_misc ADD TABLE tab_alter");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION regress_sub2_misc REFRESH PUBLICATION");
+
+$node_subscriber2->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_order" is missing replicated column: "c"/,
+ $offset2);
+
+#####################
+# TEST tab_alter
+#
+# Drop the generated column's expression on subscriber side.
+# This changes the generated column into a non-generated column.
+#####################
+
+# change a gencol to a nogen col
+$node_subscriber2->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN c DROP EXPRESSION");
+
+# insert some data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter (a) VALUES (4), (5)");
+
+# confirm that replication now works for the subscriber nogen col
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b, c FROM tab_alter ORDER BY a");
+is($result, qq(), 'after drop generated column expression');
+
+# cleanup
+$node_subscriber2->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub2_misc");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION regress_pub_misc");
+
+#####################
# try it with a subscriber-side trigger
$node_subscriber->safe_psql(
@@ -84,7 +597,7 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (7), (8)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 9 WHERE a = 7");
-$node_publisher->wait_for_catchup('sub1');
+$node_publisher->wait_for_catchup('regress_sub1_tab1');
$result =
$node_subscriber->safe_psql('postgres', "SELECT * FROM tab1 ORDER BY 1");
--
2.41.0.windows.3
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-19 05:31 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-08-19 05:31 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, Here are my review comments for v27-0001.
======
contrib/test_decoding/expected/generated_columns.out
contrib/test_decoding/sql/generated_columns.sql
+-- By default, 'include-generated-columns' is enabled, so the values
for the generated column 'b' will be replicated even if it is not
explicitly specified.
nit - The "default" is only like this for "test_decoding" (e.g., the
CREATE SUBSCRIPTION option is the opposite), so let's make the comment
clearer about that.
nit - Use sentence case in the comments.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index 48f900f..9b03f6d 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -1,13 +1,14 @@
--- test decoding of generated columns
+-- Test decoding of generated columns.
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
?column?
----------
init
(1 row)
--- column b' is a generated column
+-- Column b' is a generated column.
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- By default, 'include-generated-columns' is enabled, so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
INSERT INTO gencoltable (a) VALUES (1);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
data
@@ -17,7 +18,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(3 rows)
--- when 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
INSERT INTO gencoltable (a) VALUES (2);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
data
@@ -27,7 +28,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(3 rows)
--- when 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
INSERT INTO gencoltable (a) VALUES (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
data
@@ -37,7 +38,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(3 rows)
--- with REPLICA IDENTITY = FULL, to show old-key data includes generated columns data for updates.
+-- When REPLICA IDENTITY = FULL, show old-key data includes generated columns data for updates.
ALTER TABLE gencoltable REPLICA IDENTITY FULL;
UPDATE gencoltable SET a = 10;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index fb156c2..7b455a1 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -1,23 +1,24 @@
--- test decoding of generated columns
+-- Test decoding of generated columns.
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
--- column b' is a generated column
+-- Column b' is a generated column.
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- By default, 'include-generated-columns' is enabled, so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
INSERT INTO gencoltable (a) VALUES (1);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
--- when 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
INSERT INTO gencoltable (a) VALUES (2);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
--- when 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
INSERT INTO gencoltable (a) VALUES (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
--- with REPLICA IDENTITY = FULL, to show old-key data includes generated columns data for updates.
+-- When REPLICA IDENTITY = FULL, show old-key data includes generated columns data for updates.
ALTER TABLE gencoltable REPLICA IDENTITY FULL;
UPDATE gencoltable SET a = 10;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
Attachments:
[text/plain] PS_NITPICKS_20240819_GENCOLS_V270001.txt (5.2K, ../../CAHut+Psn4QwmX4uJxGydV4fe_thO7nOLUg61gVKfEpzDe2Jm4A@mail.gmail.com/2-PS_NITPICKS_20240819_GENCOLS_V270001.txt)
download | inline diff:
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
index 48f900f..9b03f6d 100644
--- a/contrib/test_decoding/expected/generated_columns.out
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -1,13 +1,14 @@
--- test decoding of generated columns
+-- Test decoding of generated columns.
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
?column?
----------
init
(1 row)
--- column b' is a generated column
+-- Column b' is a generated column.
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- By default, 'include-generated-columns' is enabled, so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
INSERT INTO gencoltable (a) VALUES (1);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
data
@@ -17,7 +18,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(3 rows)
--- when 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
INSERT INTO gencoltable (a) VALUES (2);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
data
@@ -27,7 +28,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(3 rows)
--- when 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
INSERT INTO gencoltable (a) VALUES (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
data
@@ -37,7 +38,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT
(3 rows)
--- with REPLICA IDENTITY = FULL, to show old-key data includes generated columns data for updates.
+-- When REPLICA IDENTITY = FULL, show old-key data includes generated columns data for updates.
ALTER TABLE gencoltable REPLICA IDENTITY FULL;
UPDATE gencoltable SET a = 10;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
index fb156c2..7b455a1 100644
--- a/contrib/test_decoding/sql/generated_columns.sql
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -1,23 +1,24 @@
--- test decoding of generated columns
+-- Test decoding of generated columns.
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
--- column b' is a generated column
+-- Column b' is a generated column.
CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
--- By default, 'include-generated-columns' is enabled, so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
INSERT INTO gencoltable (a) VALUES (1);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
--- when 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
INSERT INTO gencoltable (a) VALUES (2);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
--- when 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
INSERT INTO gencoltable (a) VALUES (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
--- with REPLICA IDENTITY = FULL, to show old-key data includes generated columns data for updates.
+-- When REPLICA IDENTITY = FULL, show old-key data includes generated columns data for updates.
ALTER TABLE gencoltable REPLICA IDENTITY FULL;
UPDATE gencoltable SET a = 10;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-19 07:10 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-08-19 07:10 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham, here are my review comments for the TAP tests patch v27-0002
======
Commit message
Tap tests for 'include-generated-columns'
~
But, it's more than that-- these are the TAP tests for all
combinations of replication related to generated columns. i.e. both
with and without 'include_generated_columns' option enabled.
======
src/test/subscription/t/011_generated.pl
I was mistaken, thinking that the v27-0002 had already been refactored
according to Vignesh's last review but it is not done yet, so I am not
going to post detailed review comments until the restructuring is
completed.
~
OTOH, there are some problems I felt have crept into v26-0001 (TAP
test is same as v27-0002), so maybe try to also take care of them (see
below) in v28-0002.
In no particular order:
* I felt it is almost useless now to have the "combo" (
"regress_pub_combo") publication. It used to have many tables when
you first created it but with every version posted it is publishing
less and less so now there are only 2 tables in it. Better to have a
specific publication for each table now and forget about "combos"
* The "TEST tab_gen_to_gen initial sync" seems to be not even checking
the table data. Why not? e.g. Even if you expect no data, you should
test for it.
* The "TEST tab_gen_to_gen replication" seems to be not even checking
the table data. Why not?
* Multiple XXX comments like "... it needs more study to determine if
the above result was actually correct, or a PG17 bug..." should be
removed. AFAIK we should well understand the expected results for all
combinations by now.
* The "TEST tab_order replication" is now getting an error saying
<missing replicated column: "c">, Now, that may now be the correct
error for this situation, but in that case, then I think the test is
not longer testing what it was intended to test (i.e. that column
order does not matter....) Probably the table definition needs
adjusting to make sure we are testing whenwe want to test, and not
just making some random scenario "PASS".
* The test "# TEST tab_alter" expected empty result also seems
unhelpful. It might be related to the previous bullet.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-22 04:52 Shubham Khanna <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 2 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-08-22 04:52 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Peter Smith <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Aug 16, 2024 at 2:47 PM vignesh C <[email protected]> wrote:
>
> On Fri, 16 Aug 2024 at 10:04, Shubham Khanna
> <[email protected]> wrote:
> >
> > On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <[email protected]> wrote:
> > >
> > > Hi Shubham,
> > >
> > > I think the v25-0001 patch only half-fixes the problems reported in my
> > > v24-0001 review.
> > >
> > > ~
> > >
> > > Background (from the commit message):
> > > This commit enables support for the 'include_generated_columns' option
> > > in logical replication, allowing the transmission of generated column
> > > information and data alongside regular table changes.
> > >
> > > ~
> > >
> > > The broken TAP test scenario in question is replicating from a
> > > "not-generated" column to a "generated" column. As the generated
> > > column is not on the publishing side, IMO the
> > > 'include_generated_columns' option should have zero effect here.
> > >
> > > In other words, I expect this TAP test for 'include_generated_columns
> > > = true' case should also be failing, as I wrote already yesterday:
> > >
> > > +# FIXME
> > > +# Since there is no generated column on the publishing side this should give
> > > +# the same result as the previous test. -- e.g. something like:
> > > +# ERROR: logical replication target relation
> > > "public.tab_nogen_to_gen" is missing
> > > +# replicated column: "b"
> >
> > I have fixed the given comments. The attached v26-0001 Patch contains
> > the required changes.
>
> Few comments:
> 1) There's no need to pass include_generated_columns in this case; we
> can retrieve it from ctx->data instead:
> @@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
> static void
> send_relation_and_attrs(Relation relation, TransactionId xid,
> LogicalDecodingContext *ctx,
> - Bitmapset *columns)
> + Bitmapset *columns,
> bool include_generated_columns)
> {
> TupleDesc desc = RelationGetDescr(relation);
> int i;
> @@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation,
> TransactionId xid,
>
> 2) Commit message:
> If the subscriber-side column is also a generated column then this option
> has no effect; the replicated data will be ignored and the subscriber
> column will be filled as normal with the subscriber-side computed or
> default data.
>
> An error will occur in this case, so the message should be updated accordingly.
>
> 3) The current test is structured as follows: a) Create all required
> tables b) Insert data into tables c) Create publications d) Create
> subscriptions e) Perform inserts and verify
> This approach can make reviewing and maintenance somewhat challenging.
>
> Instead, could you modify it to: a) Create the required table for a
> single test b) Insert data for this test c) Create the publication for
> this test d) Create the subscriptions for this test e) Perform inserts
> and verify f) Clean up
>
> 4) We can maintain the test as a separate 0002 patch, as it may need a
> few rounds of review and final adjustments. Once it's fully completed,
> we can merge it back in.
>
> 5) Once we create and drop publication/subscriptions for individual
> tests, we won't need such extensive configuration; we should be able
> to run them with default values:
> +$node_publisher->append_conf(
> + 'postgresql.conf',
> + "max_wal_senders = 20
> + max_replication_slots = 20");
Fixed all the given comments. The attached patches contain the
suggested changes.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] v28-0002-Tap-tests-for-include-generated-columns.patch (14.5K, ../../CAHv8RjL7rkxk6qSroRPg5ZARWMdK2Nd4-QyYNeoc2vhBm3cdDg@mail.gmail.com/2-v28-0002-Tap-tests-for-include-generated-columns.patch)
download | inline diff:
From 878905788911e1cf6fea0e0b24f1d688343f245b Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Fri, 16 Aug 2024 16:04:30 +0530
Subject: [PATCH v28 2/2] Tap tests for 'include-generated-columns'
Tap tests for for all combinations of replication related to generated columns,
i.e. both with and without 'include_generated_columns' option enabled.
---
src/test/subscription/t/011_generated.pl | 334 +++++++++++++++++++++--
1 file changed, 313 insertions(+), 21 deletions(-)
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index 8b2e5f4708..493c0d12dc 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -20,35 +20,29 @@ $node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED)"
-);
-
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int)"
-);
-
-# data for initial sync
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
+ INSERT INTO tab1 (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION pub1 FOR ALL TABLES;
+));
-$node_publisher->safe_psql('postgres',
- "INSERT INTO tab1 (a) VALUES (1), (2), (3)");
-
-$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
-$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
-);
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 22) STORED, c int);
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1;
+));
-# Wait for initial sync of all subscriptions
+# Wait for initial sync of node_subscriber
$node_subscriber->wait_for_subscription_sync;
+# Check initial sync data is synchronized after initial sync.
my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');
-# data to replicate
-
+# Insert data to verify incremental sync
$node_publisher->safe_psql('postgres', "INSERT INTO tab1 VALUES (4), (5)");
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
@@ -75,7 +69,7 @@ END $$;
CREATE TRIGGER test1 BEFORE INSERT OR UPDATE ON tab1
FOR EACH ROW
- EXECUTE PROCEDURE tab1_trigger_func();
+ EXECUTE PROCEDURE tab1_trigger_func();
ALTER TABLE tab1 ENABLE REPLICA TRIGGER test1;
});
@@ -96,4 +90,302 @@ is( $result, qq(1|22|
8|176|18
9|198|19), 'generated columns replicated with trigger');
+$node_subscriber->safe_psql('postgres', "CREATE DATABASE test");
+
+# =============================================================================
+# Testcase start: Publisher table with a generated column (b) and subscriber
+# table a with regular column (b).
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+ INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION regress_pub_gen_to_nogen FOR TABLE tab_gen_to_nogen;
+));
+
+# Create subscription with include_generated_columns as false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int);
+ CREATE SUBSCRIPTION regress_sub1_gen_to_nogen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_nogen WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Create subscription with include_generated_columns as true.
+$node_subscriber->safe_psql(
+ 'test', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int);
+ CREATE SUBSCRIPTION regress_sub2_gen_to_nogen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_nogen WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Verify that column 'b' is not replicated for the subscription
+# regress_sub1_gen_to_nogen when include_generated_columns is set to false
+# during initial sync.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'tab_gen_to_nogen, when include_generated_columns=false');
+
+# Insert data to verify incremental sync
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# Verify that column 'b' is not replicated for the subscription
+# regress_sub1_gen_to_nogen when include_generated_columns is set to false
+# during incremental sync.
+$node_publisher->wait_for_catchup('regress_sub1_gen_to_nogen');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'confirm generated columns are not replicated when the subscriber-side column is not generated'
+);
+
+# Verify that column 'b' is replicated for the subscription
+# regress_sub2_gen_to_nogen when include_generated_columns is set to true.
+$node_publisher->wait_for_catchup('regress_sub2_gen_to_nogen');
+$result =
+ $node_subscriber->safe_psql('test',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'confirm generated columns are replicated when the subscriber-side column is not generated'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_gen_to_nogen");
+$node_subscriber->safe_psql('test',
+ "DROP SUBSCRIPTION regress_sub2_gen_to_nogen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_gen_to_nogen");
+
+# Testcase end: Publisher table with a generated column (b) and subscriber
+# table a with regular column (b).
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Publisher table with a generated column (b) on the publisher,
+# where column (b) is not present on the subscriber.
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+ INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION regress_pub_gen_to_missing FOR TABLE tab_gen_to_missing;
+));
+
+# Create subscription with include_generated_columns as false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_missing (a int);
+ CREATE SUBSCRIPTION regress_sub1_gen_to_missing CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_missing WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Create subscription with include_generated_columns as true.
+$node_subscriber->safe_psql(
+ 'test', qq(
+ CREATE TABLE tab_gen_to_missing (a int);
+ CREATE SUBSCRIPTION regress_sub2_gen_to_missing CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_missing WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Verify that initial sync data is correctly synchronized to the subscription
+# regress_sub1_gen_to_missing with copy_data set to true after the initial sync.
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+
+# Insert data to verify incremental sync.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
+
+# Verify that column 'b' is not replicated for the subscription
+# regress_sub1_gen_to_missing when include_generated_columns is set to false.
+$node_publisher->wait_for_catchup('regress_sub1_gen_to_missing');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5), 'missing generated column, include_generated_columns = false');
+
+my $offset = -s $node_subscriber->logfile;
+
+# Verify that an error is thrown during incremental data sync for the
+# subscription regress_sub2_gen_to_missing when include_generated_columns is
+# set to true.
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_gen_to_missing");
+
+# Testcase end: Publisher table with a generated column (b) on the publisher,
+# where column (b) is not present on the subscriber.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Subscriber table with a generated column (b) on the
+# subscriber, where column (b) is not present on the publisher.
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_missing_to_gen (a int);
+ INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION regress_pub_missing_to_gen FOR TABLE tab_missing_to_gen;
+));
+
+# Create subscription with include_generated_columns as false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub1_missing_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_missing_to_gen WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Create subscription with include_generated_columns as true.
+$node_subscriber->safe_psql(
+ 'test', qq(
+ CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub2_missing_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_missing_to_gen WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Verify that initial sync data is correctly synchronized to the subscription
+# regress_sub1_gen_to_missing with copy_data set to true following the initial
+# sync.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
+);
+
+# Insert data to verify incremental sync.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# Verify that column 'b' is not replicated but is generated as usual for the
+# subscription regress_sub1_gen_to_missing when include_generated_columns is
+# set to false.
+$node_publisher->wait_for_catchup('regress_sub1_missing_to_gen');
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# Verify that column 'b' is not replicated but is generated as expected for the
+# subscription regress_sub1_gen_to_missing when include_generated_columns is set
+# to true.
+$node_publisher->wait_for_catchup('regress_sub2_missing_to_gen');
+$result =
+ $node_subscriber->safe_psql('test',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'confirm when publisher col is missing, subscriber generated columns are generated as normal'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_missing_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_missing_to_gen");
+
+# Testcase end: Subscriber table with a generated column (b) on the
+# subscriber, where column (b) is not present on the publisher.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Publisher table with a non-generated column (b) on the
+# publisher and generated column(b) on subscriber.
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_nogen_to_gen (a int, b int);
+ INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3);
+ CREATE PUBLICATION regress_pub_nogen_to_gen FOR TABLE tab_nogen_to_gen;
+));
+
+$offset = -s $node_subscriber->logfile;
+
+# Create subscription with include_generated_columns as false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub1_nogen_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Verify that an error occurs during the initial sync of data from a
+# non-generated to a generated column for the subscription
+# regress_sub1_nogen_to_gen when include_generated_columns is set to false.
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_nogen_to_gen");
+
+# Create subscription with include_generated_columns as true.
+$node_subscriber->safe_psql(
+ 'test', qq(
+ CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub2_nogen_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = true, copy_data = false);
+));
+
+$offset = -s $node_subscriber->logfile;
+
+# Verify that an error occurs during incremental sync of data from a
+# non-generated to a generated column for the subscription
+# regress_sub2_nogen_to_gen when include_generated_columns is set to true.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+
+# cleanup
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_nogen_to_gen");
+
+# Testcase end: Replication of table with non-generated column(b) on publisher
+# and generated column(b) on subscriber.
+# =============================================================================
+
done_testing();
--
2.34.1
[application/octet-stream] v28-0001-Enable-support-for-include_generated_columns-opt.patch (85.2K, ../../CAHv8RjL7rkxk6qSroRPg5ZARWMdK2Nd4-QyYNeoc2vhBm3cdDg@mail.gmail.com/3-v28-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 659beafb23fc23d68597875af2e81fa6f3bd353e Mon Sep 17 00:00:00 2001
From: Khanna <[email protected]>
Date: Fri, 16 Aug 2024 16:02:26 +0530
Subject: [PATCH v28] Enable support for 'include_generated_columns' option`
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Using Create Subscription:
CREATE SUBSCRIPTION regress_sub_combo2 CONNECTION '$publisher_connstr'
PUBLICATION regress_pub_combo WITH (include_generated_columns = true,
copy_data = false)
Currently 'copy_data' option with 'include_generated_columns' option is not
supported.
A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/generated_columns.out | 60 +++++++
contrib/test_decoding/meson.build | 1 +
.../test_decoding/sql/generated_columns.sql | 28 ++++
contrib/test_decoding/test_decoding.c | 26 ++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 +-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++--
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 34 +++-
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 11 ++
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++++++++---------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/031_column_list.pl | 6 +-
29 files changed, 397 insertions(+), 134 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..59f0956e85 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000000..9b03f6d6bd
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,60 @@
+-- Test decoding of generated columns.
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- Column b' is a generated column.
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+INSERT INTO gencoltable (a) VALUES (1);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ COMMIT
+(3 rows)
+
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+INSERT INTO gencoltable (a) VALUES (2);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ COMMIT
+(3 rows)
+
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+INSERT INTO gencoltable (a) VALUES (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:3
+ COMMIT
+(3 rows)
+
+-- When REPLICA IDENTITY = FULL, show old-key data includes generated columns data for updates.
+ALTER TABLE gencoltable REPLICA IDENTITY FULL;
+UPDATE gencoltable SET a = 10;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: UPDATE: old-key: a[integer]:1 b[integer]:2 new-tuple: a[integer]:10 b[integer]:20
+ table public.gencoltable: UPDATE: old-key: a[integer]:2 b[integer]:4 new-tuple: a[integer]:10 b[integer]:20
+ table public.gencoltable: UPDATE: old-key: a[integer]:3 b[integer]:6 new-tuple: a[integer]:10 b[integer]:20
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc81a2..718bf1b2d9 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000000..7b455a17c6
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,28 @@
+-- Test decoding of generated columns.
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- Column b' is a generated column.
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even if it is not explicitly specified.
+INSERT INTO gencoltable (a) VALUES (1);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+INSERT INTO gencoltable (a) VALUES (2);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+INSERT INTO gencoltable (a) VALUES (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+-- When REPLICA IDENTITY = FULL, show old-key data includes generated columns data for updates.
+ALTER TABLE gencoltable REPLICA IDENTITY FULL;
+UPDATE gencoltable SET a = 10;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..eaa3dbf9db 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d35514c..dced1b5026 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 11b6456779..2765fa3629 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3324,6 +3324,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>
origin
@@ -6542,8 +6553,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ee27a5873a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 7fe5fe2b86..00a66c12ce 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -511,7 +511,6 @@ pub_collist_validate(Relation targetrel, List *columns)
{
Bitmapset *set = NULL;
ListCell *lc;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
foreach(lc, columns)
{
@@ -530,12 +529,6 @@ pub_collist_validate(Relation targetrel, List *columns)
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1214,7 +1207,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..3803ce5459 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b925c464ae..27c4d43ec4 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957cd87..dc317b501a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2741..e694baca0a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 38c2895307..de4aca4e38 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4470,6 +4470,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4eaf68..55fc16dc5d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -283,11 +283,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +398,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -748,9 +760,9 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx,
- Bitmapset *columns)
+ LogicalDecodingContext *ctx, Bitmapset *columns)
{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +778,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !data->include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +797,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, data->include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1100,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1546,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b6e01d3d29..db5dd66c11 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4847,6 +4847,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4919,11 +4920,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4962,6 +4969,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -5008,6 +5016,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5254,6 +5264,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..28752ade7e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 5bcc2244d5..dde93d0406 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2983,6 +2983,17 @@ my %tests = (
like => { %full_runs, section_post_data => 1, },
},
+ 'CREATE SUBSCRIPTION sub4' => {
+ create_order => 50,
+ create_sql => 'CREATE SUBSCRIPTION sub4
+ CONNECTION \'dbname=postgres\' PUBLICATION pub1
+ WITH (connect = false, origin = any, include_generated_columns = true);',
+ regexp => qr/^
+ \QCREATE SUBSCRIPTION sub4 CONNECTION 'dbname=postgres' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub4', include_generated_columns = true);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ },
+
'ALTER PUBLICATION pub1 ADD TABLE test_table' => {
create_order => 51,
create_sql =>
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..2e8e70d4d6 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 024469474d..3c7e563807 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..37e6dd9898 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638a2e..34ec40b07e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1147..224394cb93 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789948..93b46fb01f 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 660245ed0c..11f3fcc8f9 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..3e08be39b7 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index f68a5b5986..f344eafca3 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl1 (id, ctid);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..7f7057d1b4 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5020..3bb2301b43 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
2.41.0.windows.3
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-22 04:56 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-08-22 04:56 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Aug 19, 2024 at 11:01 AM Peter Smith <[email protected]> wrote:
>
> Hi, Here are my review comments for v27-0001.
>
> ======
> contrib/test_decoding/expected/generated_columns.out
> contrib/test_decoding/sql/generated_columns.sql
>
> +-- By default, 'include-generated-columns' is enabled, so the values
> for the generated column 'b' will be replicated even if it is not
> explicitly specified.
>
> nit - The "default" is only like this for "test_decoding" (e.g., the
> CREATE SUBSCRIPTION option is the opposite), so let's make the comment
> clearer about that.
> nit - Use sentence case in the comments.
I have addressed all the comments in the v-28-0001 Patch. Please refer
to the updated v28-0001 Patch here in [1]. See [1] for the changes
added.
[1] https://www.postgresql.org/message-id/CAHv8RjL7rkxk6qSroRPg5ZARWMdK2Nd4-QyYNeoc2vhBm3cdDg%40mail.gma...
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-22 04:58 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-08-22 04:58 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Aug 19, 2024 at 12:40 PM Peter Smith <[email protected]> wrote:
>
> Hi Shubham, here are my review comments for the TAP tests patch v27-0002
>
> ======
> Commit message
>
> Tap tests for 'include-generated-columns'
>
> ~
>
> But, it's more than that-- these are the TAP tests for all
> combinations of replication related to generated columns. i.e. both
> with and without 'include_generated_columns' option enabled.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> I was mistaken, thinking that the v27-0002 had already been refactored
> according to Vignesh's last review but it is not done yet, so I am not
> going to post detailed review comments until the restructuring is
> completed.
>
> ~
>
> OTOH, there are some problems I felt have crept into v26-0001 (TAP
> test is same as v27-0002), so maybe try to also take care of them (see
> below) in v28-0002.
>
> In no particular order:
>
> * I felt it is almost useless now to have the "combo" (
> "regress_pub_combo") publication. It used to have many tables when
> you first created it but with every version posted it is publishing
> less and less so now there are only 2 tables in it. Better to have a
> specific publication for each table now and forget about "combos"
>
> * The "TEST tab_gen_to_gen initial sync" seems to be not even checking
> the table data. Why not? e.g. Even if you expect no data, you should
> test for it.
>
> * The "TEST tab_gen_to_gen replication" seems to be not even checking
> the table data. Why not?
>
> * Multiple XXX comments like "... it needs more study to determine if
> the above result was actually correct, or a PG17 bug..." should be
> removed. AFAIK we should well understand the expected results for all
> combinations by now.
>
> * The "TEST tab_order replication" is now getting an error saying
> <missing replicated column: "c">, Now, that may now be the correct
> error for this situation, but in that case, then I think the test is
> not longer testing what it was intended to test (i.e. that column
> order does not matter....) Probably the table definition needs
> adjusting to make sure we are testing whenwe want to test, and not
> just making some random scenario "PASS".
>
> * The test "# TEST tab_alter" expected empty result also seems
> unhelpful. It might be related to the previous bullet.
I have addressed all the comments in the v-28-0002 Patch. Please refer
to the updated v28-0002 Patch here in [1]. See [1] for the changes
added.
[1] https://www.postgresql.org/message-id/CAHv8RjL7rkxk6qSroRPg5ZARWMdK2Nd4-QyYNeoc2vhBm3cdDg%40mail.gma...
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-22 11:58 vignesh C <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 0 replies; 60+ messages in thread
From: vignesh C @ 2024-08-22 11:58 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Peter Smith <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, 22 Aug 2024 at 10:22, Shubham Khanna
<[email protected]> wrote:
>
> On Fri, Aug 16, 2024 at 2:47 PM vignesh C <[email protected]> wrote:
> >
> > On Fri, 16 Aug 2024 at 10:04, Shubham Khanna
> > <[email protected]> wrote:
> > >
> > > On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <[email protected]> wrote:
> > > >
> > > > Hi Shubham,
> > > >
> > > > I think the v25-0001 patch only half-fixes the problems reported in my
> > > > v24-0001 review.
> > > >
> > > > ~
> > > >
> > > > Background (from the commit message):
> > > > This commit enables support for the 'include_generated_columns' option
> > > > in logical replication, allowing the transmission of generated column
> > > > information and data alongside regular table changes.
> > > >
> > > > ~
> > > >
> > > > The broken TAP test scenario in question is replicating from a
> > > > "not-generated" column to a "generated" column. As the generated
> > > > column is not on the publishing side, IMO the
> > > > 'include_generated_columns' option should have zero effect here.
> > > >
> > > > In other words, I expect this TAP test for 'include_generated_columns
> > > > = true' case should also be failing, as I wrote already yesterday:
> > > >
> > > > +# FIXME
> > > > +# Since there is no generated column on the publishing side this should give
> > > > +# the same result as the previous test. -- e.g. something like:
> > > > +# ERROR: logical replication target relation
> > > > "public.tab_nogen_to_gen" is missing
> > > > +# replicated column: "b"
> > >
> > > I have fixed the given comments. The attached v26-0001 Patch contains
> > > the required changes.
> >
> > Few comments:
> > 1) There's no need to pass include_generated_columns in this case; we
> > can retrieve it from ctx->data instead:
> > @@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
> > static void
> > send_relation_and_attrs(Relation relation, TransactionId xid,
> > LogicalDecodingContext *ctx,
> > - Bitmapset *columns)
> > + Bitmapset *columns,
> > bool include_generated_columns)
> > {
> > TupleDesc desc = RelationGetDescr(relation);
> > int i;
> > @@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation,
> > TransactionId xid,
> >
> > 2) Commit message:
> > If the subscriber-side column is also a generated column then this option
> > has no effect; the replicated data will be ignored and the subscriber
> > column will be filled as normal with the subscriber-side computed or
> > default data.
> >
> > An error will occur in this case, so the message should be updated accordingly.
> >
> > 3) The current test is structured as follows: a) Create all required
> > tables b) Insert data into tables c) Create publications d) Create
> > subscriptions e) Perform inserts and verify
> > This approach can make reviewing and maintenance somewhat challenging.
> >
> > Instead, could you modify it to: a) Create the required table for a
> > single test b) Insert data for this test c) Create the publication for
> > this test d) Create the subscriptions for this test e) Perform inserts
> > and verify f) Clean up
> >
> > 4) We can maintain the test as a separate 0002 patch, as it may need a
> > few rounds of review and final adjustments. Once it's fully completed,
> > we can merge it back in.
> >
> > 5) Once we create and drop publication/subscriptions for individual
> > tests, we won't need such extensive configuration; we should be able
> > to run them with default values:
> > +$node_publisher->append_conf(
> > + 'postgresql.conf',
> > + "max_wal_senders = 20
> > + max_replication_slots = 20");
>
> Fixed all the given comments. The attached patches contain the
> suggested changes.
Few comments:
1) This is already been covered in the first existing test case, may
be this can be removed:
# =============================================================================
# Testcase start: Subscriber table with a generated column (b) on the
# subscriber, where column (b) is not present on the publisher.
This existing test:
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
INSERT INTO tab1 (a) VALUES (1), (2), (3);
CREATE PUBLICATION pub1 FOR ALL TABLES;
));
$node_subscriber->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a *
22) STORED, c int);
CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1;
));
2) Can we have this test verified with include_generated_columns =
true too like how others are done:
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
INSERT INTO tab1 (a) VALUES (1), (2), (3);
CREATE PUBLICATION pub1 FOR ALL TABLES;
));
$node_subscriber->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a *
22) STORED, c int);
CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1;
));
3) There is a typo in this comment:
3.a) # Testcase start: Publisher table with a generated column (b)
and subscriber
# table a with regular column (b).
It should be:
# Testcase start: Publisher table with a generated column (b) and subscriber
# table with a regular column (b).
3.b) similarly here too:
# Testcase end: Publisher table with a generated column (b) and subscriber
# table a with regular column (b).
3.c) The comments are not consistent, sometimes mentioned as
column(b) and sometimes as column (b). We can keep it consistent.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-08-23 10:56 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 0 replies; 60+ messages in thread
From: Peter Smith @ 2024-08-23 10:56 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi Shubham,
I have reviewed v28* and posted updated v29 versions of patches 0001 and 0002.
If you are OK with these changes, the next task would be to pg_indent
them, then rebase the remaining patches (0003 etc.) and include those
with the next patchset version.
//////////
Patch v29-0001 changes:
nit - Fixed typo in comments.
nit - Removed an unnecessary format change for the unchanged
send_relation_and_attrs declaration.
//////////
Patch v29-0002 changes:
1.
Made fixes to address Vignesh's review comments [1].
2.
Added the missing test cases for tab_gen_to_gen, and tab_alter.
3.
Multiple other modifications include:
nit - Renamed the test database /test/test_igc_true/ because 'test'
was too vague.
nit - This patch does not need to change most of the existing 'tab1'
test. So we should not be reformating the existing test code for no
reason.
nit - I added a summary comment to describe the test combinations
nit - The "Testcase end" comments were unnecessary and prone to error,
so I removed them.
nit - Change comments /incremental sync/incremental replication/
nit - Added XXX notes about copy_data=false. These are reminders for
to change code in later TAP patches
nit - Rearranged test steps so the publisher does not do incremental
INSERT until all initial sync tests are done
nit - Added initial sync tests even if copy_data=false. This is for
completeness - these will be handled in a later TAP patch
nit - The table names are self-explanatory, so some of the test
"messages" were simplified
======
[1] https://www.postgresql.org/message-id/CALDaNm31LZQfeR8Vv1qNCOREGffvZbgGDrTp%3D3h%3DEHiHTEO2pQ%40mail...
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
[application/octet-stream] v29-0001-Enable-support-for-include_generated_columns-opt.patch (85.1K, ../../CAHut+Pt_HXOn1NNiiZvCMz6gGXEg04gAANVBOwHUyyRmshWZJA@mail.gmail.com/2-v29-0001-Enable-support-for-include_generated_columns-opt.patch)
download | inline diff:
From 113519d3afc83c89efe6d21fc18e5114e2a8fade Mon Sep 17 00:00:00 2001
From: Peter Smith <[email protected]>
Date: Fri, 23 Aug 2024 10:03:38 +1000
Subject: [PATCH v29] Enable support for 'include_generated_columns' option.
Currently generated column values are not replicated because it is assumed
that the corresponding subscriber-side table will generate its own values
for those columns.
This commit enables support for the 'include_generated_columns' option in
logical replication, allowing the transmission of generated column information
and data alongside regular table changes.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.
Example usage for test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL,
'include-xids', '0','skip-empty-xacts', '1',
'include-generated-columns','1');
Example usage of subscription option:
CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr'
PUBLICATION pub1 WITH (include_generated_columns = true,
copy_data = false)
The 'copy_data' option with 'include_generated_columns' option is not
currently supported. A future patch will remove this limitation.
'include_generated_columns' cannot be altered as it can lead to inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../test_decoding/expected/generated_columns.out | 60 ++++++++
contrib/test_decoding/meson.build | 1 +
contrib/test_decoding/sql/generated_columns.sql | 28 ++++
contrib/test_decoding/test_decoding.c | 26 +++-
doc/src/sgml/ddl.sgml | 6 +-
doc/src/sgml/protocol.sgml | 17 ++-
doc/src/sgml/ref/create_subscription.sgml | 20 +++
src/backend/catalog/pg_publication.c | 9 +-
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 31 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 4 +
src/backend/replication/logical/proto.c | 56 +++++---
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 31 +++-
src/bin/pg_dump/pg_dump.c | 17 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 11 ++
src/bin/psql/describe.c | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/logicalproto.h | 13 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 2 +
src/test/regress/expected/publication.out | 4 +-
src/test/regress/expected/subscription.out | 157 +++++++++++----------
src/test/regress/sql/publication.sql | 3 +-
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/031_column_list.pl | 6 +-
29 files changed, 396 insertions(+), 132 deletions(-)
create mode 100644 contrib/test_decoding/expected/generated_columns.out
create mode 100644 contrib/test_decoding/sql/generated_columns.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a5..59f0956 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ generated_columns
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/generated_columns.out b/contrib/test_decoding/expected/generated_columns.out
new file mode 100644
index 0000000..d6a402c
--- /dev/null
+++ b/contrib/test_decoding/expected/generated_columns.out
@@ -0,0 +1,60 @@
+-- Test decoding of generated columns.
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- Column 'b' is a generated column.
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even when the parameter is not specified.
+INSERT INTO gencoltable (a) VALUES (1);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ COMMIT
+(3 rows)
+
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+INSERT INTO gencoltable (a) VALUES (2);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ COMMIT
+(3 rows)
+
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+INSERT INTO gencoltable (a) VALUES (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+ data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:3
+ COMMIT
+(3 rows)
+
+-- When REPLICA IDENTITY is FULL, the old-key data includes the generated columns data for updates.
+ALTER TABLE gencoltable REPLICA IDENTITY FULL;
+UPDATE gencoltable SET a = 10;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+ data
+-------------------------------------------------------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: UPDATE: old-key: a[integer]:1 b[integer]:2 new-tuple: a[integer]:10 b[integer]:20
+ table public.gencoltable: UPDATE: old-key: a[integer]:2 b[integer]:4 new-tuple: a[integer]:10 b[integer]:20
+ table public.gencoltable: UPDATE: old-key: a[integer]:3 b[integer]:6 new-tuple: a[integer]:10 b[integer]:20
+ COMMIT
+(5 rows)
+
+DROP TABLE gencoltable;
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+ ?column?
+----------
+ stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f643dc8..718bf1b 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'generated_columns',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/generated_columns.sql b/contrib/test_decoding/sql/generated_columns.sql
new file mode 100644
index 0000000..be04db4
--- /dev/null
+++ b/contrib/test_decoding/sql/generated_columns.sql
@@ -0,0 +1,28 @@
+-- Test decoding of generated columns.
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- Column 'b' is a generated column.
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- For 'test_decoding' the parameter 'include-generated-columns' is enabled by default,
+-- so the values for the generated column 'b' will be replicated even when the parameter is not specified.
+INSERT INTO gencoltable (a) VALUES (1);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- When 'include-generated-columns' is enabled, the values of the generated column 'b' will be replicated.
+INSERT INTO gencoltable (a) VALUES (2);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+-- When 'include-generated-columns' is disabled, the values of the generated column 'b' will not be replicated.
+INSERT INTO gencoltable (a) VALUES (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '0');
+
+-- When REPLICA IDENTITY is FULL, the old-key data includes the generated columns data for updates.
+ALTER TABLE gencoltable REPLICA IDENTITY FULL;
+UPDATE gencoltable SET a = 10;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13..eaa3dbf 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -31,6 +31,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_generated_columns;
} TestDecodingData;
/*
@@ -168,6 +169,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_timestamp = false;
data->skip_empty_xacts = false;
data->only_local = false;
+ data->include_generated_columns = true;
ctx->output_plugin_private = data;
@@ -259,6 +261,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;
+ else if (!parse_bool(strVal(elem->arg), &data->include_generated_columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname));
+ }
else
{
ereport(ERROR,
@@ -521,7 +533,8 @@ print_literal(StringInfo s, Oid typid, char *outputstr)
/* print the tuple 'tuple' into the StringInfo s */
static void
-tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_nulls)
+tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple,
+ bool skip_nulls, bool include_generated_columns)
{
int natt;
@@ -544,6 +557,9 @@ tuple_to_stringinfo(StringInfo s, TupleDesc tupdesc, HeapTuple tuple, bool skip_
if (attr->attisdropped)
continue;
+ if (attr->attgenerated && !include_generated_columns)
+ continue;
+
/*
* Don't print system columns, oid will already have been printed if
* present.
@@ -641,7 +657,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
appendStringInfoString(ctx->out, " UPDATE:");
@@ -650,7 +666,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out, " old-key:");
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
appendStringInfoString(ctx->out, " new-tuple:");
}
@@ -659,7 +675,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.newtuple,
- false);
+ false, data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
appendStringInfoString(ctx->out, " DELETE:");
@@ -671,7 +687,7 @@ pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
else
tuple_to_stringinfo(ctx->out, tupdesc,
change->data.tp.oldtuple,
- true);
+ true, data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 626d355..dced1b5 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
</listitem>
<listitem>
<para>
- Generated columns are skipped for logical replication and cannot be
- specified in a <command>CREATE PUBLICATION</command> column list.
+ Generated columns may be skipped during logical replication according to the
+ <command>CREATE SUBSCRIPTION</command> option
+ <link linkend="sql-createsubscription-params-with-include-generated-columns">
+ <literal>include_generated_columns</literal></link>.
</para>
</listitem>
</itemizedlist>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 11b6456..2765fa3 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3325,6 +3325,17 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</varlistentry>
<varlistentry>
+ <term>include_generated_columns</term>
+ <listitem>
+ <para>
+ Boolean option to enable generated columns. This option controls
+ whether generated columns should be included in the string
+ representation of tuples during logical decoding in PostgreSQL.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term>
origin
</term>
@@ -6542,8 +6553,10 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</variablelist>
<para>
- Next, the following message part appears for each column included in
- the publication (except generated columns):
+ Next, the following message parts appear for each column included in
+ the publication (generated columns are excluded unless the parameter
+ <link linkend="protocol-logical-replication-params">
+ <literal>include_generated_columns</literal></link> specifies otherwise):
</para>
<variablelist>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d94..ee27a58 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,26 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-include-generated-columns">
+ <term><literal>include_generated_columns</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the generated columns present in the tables
+ associated with the subscription should be replicated.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ If the subscriber-side column is also a generated column then this option
+ has no effect; the subscriber column will be filled as normal with the
+ subscriber-side computed or default data.
+ </para>
+ <para>
+ This parameter can only be set <literal>true</literal> if <literal>copy_data</literal> is
+ set to <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 7fe5fe2..00a66c1 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -511,7 +511,6 @@ pub_collist_validate(Relation targetrel, List *columns)
{
Bitmapset *set = NULL;
ListCell *lc;
- TupleDesc tupdesc = RelationGetDescr(targetrel);
foreach(lc, columns)
{
@@ -530,12 +529,6 @@ pub_collist_validate(Relation targetrel, List *columns)
errmsg("cannot use system column \"%s\" in publication column list",
colname));
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
- colname));
-
if (bms_is_member(attnum, set))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -1214,7 +1207,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
attnums[nattnums++] = att->attnum;
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc915..3803ce5 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->includegencols = subform->subincludegencols;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b925c46..27c4d43 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -73,6 +73,7 @@
#define SUBOPT_FAILOVER 0x00002000
#define SUBOPT_LSN 0x00004000
#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMNS 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -100,6 +101,7 @@ typedef struct SubOpts
bool failover;
char *origin;
XLogRecPtr lsn;
+ bool include_generated_columns;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -164,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->failover = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
+ if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ opts->include_generated_columns = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -357,6 +361,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_LSN;
opts->lsn = lsn;
}
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_INCLUDE_GENERATED_COLUMNS))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMNS;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -437,6 +450,20 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_columns are true. COPY of generated columns is not
+ * supported yet.
+ */
+ if (opts->copy_data && opts->include_generated_columns)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_columns = true"));
+ }
}
/*
@@ -594,7 +621,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+ SUBOPT_INCLUDE_GENERATED_COLUMNS);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -714,6 +742,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
publicationListToArray(publications);
values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(opts.origin);
+ values[Anum_pg_subscription_subincludegencols - 1] = BoolGetDatum(opts.include_generated_columns);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 97f957c..dc317b5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -598,6 +598,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", origin '%s'",
options->proto.logical.origin);
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 180000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 980f6e2..e694bac 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -30,10 +30,12 @@
#define TRUNCATE_RESTART_SEQS (1<<1)
static void logicalrep_write_attrs(StringInfo out, Relation rel,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -412,7 +414,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -424,7 +427,8 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -457,7 +461,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -478,11 +483,13 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary, columns);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -532,7 +539,7 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
void
logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, bool binary,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -552,7 +559,8 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary, columns);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns,
+ include_generated_columns);
}
/*
@@ -668,7 +676,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
- Bitmapset *columns)
+ Bitmapset *columns, bool include_generated_columns)
{
char *relname;
@@ -690,7 +698,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel, columns);
+ logicalrep_write_attrs(out, rel, columns, include_generated_columns);
}
/*
@@ -767,7 +775,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary, Bitmapset *columns)
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
Datum *values;
@@ -781,7 +790,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -802,7 +814,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
Form_pg_type typclass;
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -923,7 +938,8 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns,
+ bool include_generated_columns)
{
TupleDesc desc;
int i;
@@ -938,7 +954,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
@@ -959,7 +978,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
Form_pg_attribute att = TupleDescAttr(desc, i);
uint8 flags = 0;
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !include_generated_columns)
continue;
if (!column_in_column_list(att->attnum, columns))
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 38c2895..de4aca4 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -4470,6 +4470,7 @@ set_stream_options(WalRcvStreamOptions *options,
options->proto.logical.twophase = false;
options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.include_generated_columns = MySubscription->includegencols;
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abef4ea..52ac64a 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -283,11 +283,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
+ bool include_generated_columns_option_given = false;
data->binary = false;
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
+ data->include_generated_columns = false;
foreach(lc, options)
{
@@ -396,6 +398,16 @@ parse_output_parameters(List *options, PGOutputData *data)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
+ else if (strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (include_generated_columns_option_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options"));
+ include_generated_columns_option_given = true;
+
+ data->include_generated_columns = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -751,6 +763,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx,
Bitmapset *columns)
{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -766,7 +779,10 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !data->include_generated_columns)
continue;
if (att->atttypid < FirstGenbkiObjectId)
@@ -782,7 +798,7 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation, columns);
+ logicalrep_write_rel(ctx->out, xid, relation, columns, data->include_generated_columns);
OutputPluginWrite(ctx, false);
}
@@ -1085,7 +1101,7 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
{
Form_pg_attribute att = TupleDescAttr(desc, i);
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
continue;
nliveatts++;
@@ -1531,15 +1547,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel, old_slot,
- new_slot, data->binary, relentry->columns);
+ new_slot, data->binary, relentry->columns,
+ data->include_generated_columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel, old_slot,
- data->binary, relentry->columns);
+ data->binary, relentry->columns,
+ data->include_generated_columns);
break;
default:
Assert(false);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b6e01d3..db5dd66 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4847,6 +4847,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subincludegencols;
int i,
ntups;
@@ -4919,11 +4920,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 180000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4962,6 +4969,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subincludegencols = PQfnumber(res, "subincludegencols");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -5008,6 +5016,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subincludegencols =
+ pg_strdup(PQgetvalue(res, i, i_subincludegencols));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5254,6 +5264,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ if (strcmp(subinfo->subincludegencols, "t") == 0)
+ appendPQExpBufferStr(query, ", include_generated_columns = true");
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e587..28752ad 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subincludegencols;
} SubscriptionInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 5bcc224..dde93d0 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2983,6 +2983,17 @@ my %tests = (
like => { %full_runs, section_post_data => 1, },
},
+ 'CREATE SUBSCRIPTION sub4' => {
+ create_order => 50,
+ create_sql => 'CREATE SUBSCRIPTION sub4
+ CONNECTION \'dbname=postgres\' PUBLICATION pub1
+ WITH (connect = false, origin = any, include_generated_columns = true);',
+ regexp => qr/^
+ \QCREATE SUBSCRIPTION sub4 CONNECTION 'dbname=postgres' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub4', include_generated_columns = true);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ },
+
'ALTER PUBLICATION pub1 ADD TABLE test_table' => {
create_order => 51,
create_sql =>
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f2..2e8e70d 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6608,6 +6608,12 @@ describeSubscriptions(const char *pattern, bool verbose)
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("Include generated columns"));
+
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
", subconninfo AS \"%s\"\n",
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0d25981..08ffd6a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3357,7 +3357,8 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
+ "disable_on_error", "enabled", "failover",
+ "include_generated_columns", "origin",
"password_required", "run_as_owner", "slot_name",
"streaming", "synchronous_commit", "two_phase");
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec..37e6dd9 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subincludegencols; /* True if generated columns should be
+ * published */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +160,7 @@ typedef struct Subscription
List *publications; /* List of publication names to subscribe to */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool includegencols; /* Publish generated columns */
} Subscription;
/* Disallow streaming in-progress transactions. */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index c409638..34ec40b 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -225,18 +225,22 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary, Bitmapset *columns);
+ TupleTableSlot *newslot, bool binary,
+ Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
Relation rel, TupleTableSlot *oldslot,
- bool binary, Bitmapset *columns);
+ bool binary, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
@@ -247,7 +251,8 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel, Bitmapset *columns);
+ Relation rel, Bitmapset *columns,
+ bool include_generated_columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 89f94e1..224394c 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -33,6 +33,7 @@ typedef struct PGOutputData
bool messages;
bool two_phase;
bool publish_no_origin;
+ bool include_generated_columns;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 132e789..93b46fb 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -186,6 +186,8 @@ typedef struct
* prepare time */
char *origin; /* Only publish data originating from the
* specified origin */
+ bool include_generated_columns; /* Publish generated
+ * columns */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 660245e..11f3fcc 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -687,9 +687,9 @@ UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR: cannot use generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot use system column "ctid" in publication column list
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1..3e08be3 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -99,6 +99,11 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
ERROR: subscription with slot_name = NONE must also set create_slot = false
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
ERROR: subscription with slot_name = NONE must also set enabled = false
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+ERROR: copy_data = true and include_generated_columns = true are mutually exclusive options
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
+ERROR: include_generated_columns requires a Boolean value
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
WARNING: subscription was created, but is not connected
@@ -116,18 +121,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +150,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +162,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +181,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +193,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +228,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +260,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +284,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +337,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +376,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +398,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +414,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Include generated columns | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+---------------------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index f68a5b5..f344eaf 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -413,8 +413,9 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl1 (id, ctid);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7..7f7057d 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -59,6 +59,10 @@ CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PU
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, enabled = false);
CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, create_slot = false);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (include_generated_columns = true, copy_data = true);
+
+-- fail - include_generated_columns must be boolean
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, include_generated_columns = foo);
-- ok - with slot_name = NONE
CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false);
diff --git a/src/test/subscription/t/031_column_list.pl b/src/test/subscription/t/031_column_list.pl
index 9a97fa5..3bb2301 100644
--- a/src/test/subscription/t/031_column_list.pl
+++ b/src/test/subscription/t/031_column_list.pl
@@ -1202,16 +1202,16 @@ $result = $node_publisher->safe_psql(
is( $result, qq(t
t), 'check the number of columns in the old tuple');
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
# So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
# list) are considered to have the same column list.
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
ALTER TABLE test_mix_4 DROP COLUMN c;
- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b, d);
CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
-- initial data
--
1.8.3.1
[application/octet-stream] v29-0002-Tap-tests-for-generated-columns.patch (17.1K, ../../CAHut+Pt_HXOn1NNiiZvCMz6gGXEg04gAANVBOwHUyyRmshWZJA@mail.gmail.com/3-v29-0002-Tap-tests-for-generated-columns.patch)
download | inline diff:
From d2103076db02c775f576ce550dc405bc699bc5aa Mon Sep 17 00:00:00 2001
From: Peter Smith <[email protected]>
Date: Fri, 23 Aug 2024 20:39:57 +1000
Subject: [PATCH v29] Tap tests for generated columns
Add tests for all combinations of generated column replication.
Also test effect of 'include_generated_columns' option true/false.
Author: Shubham Khanna, Peter Smith
Reviewed-by: Vignesh C
---
src/test/subscription/t/011_generated.pl | 413 +++++++++++++++++++++++++++++++
1 file changed, 413 insertions(+)
mode change 100644 => 100755 src/test/subscription/t/011_generated.pl
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
old mode 100644
new mode 100755
index 8b2e5f4..d4f11cf
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -96,4 +96,417 @@ is( $result, qq(1|22|
8|176|18
9|198|19), 'generated columns replicated with trigger');
+# =============================================================================
+# The following test cases exercise logical replication for all combinations
+# where there is a generated column on one or both sides of pub/sub:
+# - generated -> generated
+# - generated -> normal
+# - generated -> missing
+# - missing -> generated
+# - normal -> generated
+#
+# Furthermore, all combinations are tested for include_generated_columns=false
+# (see subscription sub1 of database 'postgres'), and
+# include_generated_columns=true (see subscription sub2 of database
+# 'test_igc_true').
+# =============================================================================
+
+$node_subscriber->safe_psql('postgres', "CREATE DATABASE test_igc_true");
+
+# --------------------------------------------------
+# Testcase: generated -> generated
+# Publisher table has generated column 'b'.
+# Subscriber table has normal column 'b'.
+# --------------------------------------------------
+
+# Note, table 'tab_gen_to_gen' is essentially same as the 'tab1' table above.
+# Since 'tab1' is already testing the include_generated_columns=false case,
+# here we need only test the include_generated_columns=true case.
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+ INSERT INTO tab_gen_to_gen (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION regress_pub_gen_to_gen FOR TABLE tab_gen_to_gen;
+));
+
+# Create subscription with include_generated_columns=true.
+# XXX copy_data=false for now. This will be changed later.
+$node_subscriber->safe_psql(
+ 'test_igc_true', qq(
+ CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub2_gen_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_gen WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Initial sync test when include_generated_columns=true.
+# XXX copy_data=false for now, so there is no initial copy. This will be changed later.
+$result = $node_subscriber->safe_psql('test_igc_true',
+ "SELECT a, b FROM tab_gen_to_gen");
+is( $result, qq(),
+ 'tab_gen_to_gen initial sync, when include_generated_columns=true');
+
+# Insert data to verify incremental replication.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_gen VALUES (4), (5)");
+
+# Verify that an error occurs. This behaviour is same as tab_nogen_to_gen.
+my $offset = -s $node_subscriber->logfile;
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_gen_to_gen" is missing replicated column: "b"/,
+ $offset);
+
+# cleanup
+$node_subscriber->safe_psql('test_igc_true',
+ "DROP SUBSCRIPTION regress_sub2_gen_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_gen_to_gen");
+
+# --------------------------------------------------
+# Testcase: generated -> normal
+# Publisher table has generated column 'b'.
+# Subscriber table has normal column 'b'.
+# --------------------------------------------------
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+ INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION regress_pub_gen_to_nogen FOR TABLE tab_gen_to_nogen;
+));
+
+# Create subscription with include_generated_columns=false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int);
+ CREATE SUBSCRIPTION regress_sub1_gen_to_nogen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_nogen WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Create subscription with include_generated_columns=true.
+# XXX copy_data=false for now. This will be changed later.
+$node_subscriber->safe_psql(
+ 'test_igc_true', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int);
+ CREATE SUBSCRIPTION regress_sub2_gen_to_nogen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_nogen WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Initial sync test when include_generated_columns=false.
+# Verify that column 'b' is not replicated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'tab_gen_to_nogen initial sync, when include_generated_columns=false');
+
+# Initial sync test when include_generated_columns=true.
+# XXX copy_data=false for now. This will be changed later.
+$result = $node_subscriber->safe_psql('test_igc_true',
+ "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(),
+ 'tab_gen_to_nogen initial sync, when include_generated_columns=true');
+
+# Insert data to verify incremental replication
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_nogen VALUES (4), (5)");
+
+# Incremental replication test when include_generated_columns=false.
+# Verify that column 'b' is not replicated.
+$node_publisher->wait_for_catchup('regress_sub1_gen_to_nogen');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|
+5|),
+ 'tab_gen_to_nogen incremental replication, when include_generated_columns=false'
+);
+
+# Incremental replication test when include_generated_columns=true.
+# Verify that column 'b' is replicated.
+$node_publisher->wait_for_catchup('regress_sub2_gen_to_nogen');
+$result = $node_subscriber->safe_psql('test_igc_true',
+ "SELECT a, b FROM tab_gen_to_nogen ORDER BY a");
+is( $result, qq(4|8
+5|10),
+ 'tab_gen_to_nogen incremental replication, when include_generated_columns=true'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_gen_to_nogen");
+$node_subscriber->safe_psql('test_igc_true',
+ "DROP SUBSCRIPTION regress_sub2_gen_to_nogen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_gen_to_nogen");
+
+# --------------------------------------------------
+# Testcase: generated -> missing
+# Publisher table has generated column 'b'.
+# Subscriber table does not have a column 'b'.
+# --------------------------------------------------
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_missing (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+ INSERT INTO tab_gen_to_missing (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION regress_pub_gen_to_missing FOR TABLE tab_gen_to_missing;
+));
+
+# Create subscription with include_generated_columns=false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_missing (a int);
+ CREATE SUBSCRIPTION regress_sub1_gen_to_missing CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_missing WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Create subscription with include_generated_columns=true.
+$node_subscriber->safe_psql(
+ 'test_igc_true', qq(
+ CREATE TABLE tab_gen_to_missing (a int);
+ CREATE SUBSCRIPTION regress_sub2_gen_to_missing CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_gen_to_missing WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Initial sync test when include_generated_columns=false.
+# Verify generated columns are not replicated, so it is otherwise a normal data sync
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(1
+2
+3), 'tab_gen_to_missing initial sync, when include_generated_columns=false');
+
+# Initial sync test when include_generate_columns=true.
+# XXX copy_data=false for now, so there is no initial copy. This will be changed later.
+$result = $node_subscriber->safe_psql('test_igc_true',
+ "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(),
+ 'tab_gen_to_missing initial sync, when include_generated_columns=true');
+
+# Insert data to verify incremental replication.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_gen_to_missing VALUES (4), (5)");
+
+# Incremental replication test when include_generated_columns=false.
+# Verify no error happens due to the missing column 'b'.
+$node_publisher->wait_for_catchup('regress_sub1_gen_to_missing');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_gen_to_missing ORDER BY a");
+is( $result, qq(1
+2
+3
+4
+5), 'tab_gen_to_missing incremental replication, when include_generated_columns=false');
+
+# Incremental replication test when include_generated_columns=true.
+# Verify that an error is thrown due to the missing column 'b'.
+$offset = -s $node_subscriber->logfile;
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]+:)? logical replication target relation "public.tab_gen_to_missing" is missing replicated column: "b"/,
+ $offset);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_gen_to_missing");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_gen_to_missing");
+
+# --------------------------------------------------
+# Testcase: missing -> generated
+# Publisher table has does not have a column 'b'.
+# Subscriber table has generated column 'b'.
+# --------------------------------------------------
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_missing_to_gen (a int);
+ INSERT INTO tab_missing_to_gen (a) VALUES (1), (2), (3);
+ CREATE PUBLICATION regress_pub_missing_to_gen FOR TABLE tab_missing_to_gen;
+));
+
+# Create subscription with include_generated_columns=false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub1_missing_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_missing_to_gen WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Create subscription with include_generated_columns=true.
+$node_subscriber->safe_psql(
+ 'test_igc_true', qq(
+ CREATE TABLE tab_missing_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub2_missing_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_missing_to_gen WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Initial sync test when include_generated_columns=false.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen");
+is( $result, qq(1|22
+2|44
+3|66), 'tab_missing_to_gen initial sync, when include_generated_columns=false'
+);
+
+# Initial sync test when include_generate_columns=true.
+# XXX copy_data=false for now, so there is no initial copy. This will be changed later.
+$result = $node_subscriber->safe_psql('test_igc_true',
+ "SELECT a FROM tab_gen_to_missing");
+is( $result, qq(),
+ 'tab_missing_to_gen initial sync, when include_generated_columns=true');
+
+# Insert data to verify incremental replication.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_missing_to_gen VALUES (4), (5)");
+
+# Incremental replication test when include_generated_columns=false.
+# Verify that column 'b' is not replicated. Subscriber generated value is used.
+$node_publisher->wait_for_catchup('regress_sub1_missing_to_gen');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(1|22
+2|44
+3|66
+4|88
+5|110),
+ 'tab_missing_to_gen incremental replication, when include_generated_columns=false'
+);
+
+# Incremental replication test when include_generated_columns=true.
+# Verify that column 'b' is not replicated. Subscriber generated value is used.
+$node_publisher->wait_for_catchup('regress_sub2_missing_to_gen');
+$result = $node_subscriber->safe_psql('test_igc_true',
+ "SELECT a, b FROM tab_missing_to_gen ORDER BY a");
+is( $result, qq(4|88
+5|110),
+ 'tab_missing_to_gen incremental replication, when include_generated_columns=true'
+);
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_missing_to_gen");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_missing_to_gen");
+
+# --------------------------------------------------
+# Testcase: normal -> generated
+# Publisher table has normal column 'b'.
+# Subscriber table has generated column 'b'.
+# --------------------------------------------------
+
+# Create table and publication.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_nogen_to_gen (a int, b int);
+ INSERT INTO tab_nogen_to_gen (a, b) VALUES (1, 1), (2, 2), (3, 3);
+ CREATE PUBLICATION regress_pub_nogen_to_gen FOR TABLE tab_nogen_to_gen;
+));
+
+# Create subscription with include_generated_columns=false.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub1_nogen_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = false, copy_data = true);
+));
+
+# Verify that an error occurs, then drop the failing subscription.
+$offset = -s $node_subscriber->logfile;
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub1_nogen_to_gen");
+
+# Create subscription with include_generated_columns=true.
+$node_subscriber->safe_psql(
+ 'test_igc_true', qq(
+ CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub2_nogen_to_gen CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_nogen_to_gen WITH (include_generated_columns = true, copy_data = false);
+));
+
+# Verify that an error occurs, then drop the failing subscription
+$offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_nogen_to_gen VALUES (4), (5)");
+$node_subscriber->wait_for_log(
+ qr/ERROR: ( [A-Z0-9]:)? logical replication target relation "public.tab_nogen_to_gen" is missing replicated column: "b"/,
+ $offset);
+$node_subscriber->safe_psql('test_igc_true',
+ "DROP SUBSCRIPTION regress_sub2_nogen_to_gen");
+
+# cleanup
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_nogen_to_gen");
+
+# =============================================================================
+# Misc test.
+#
+# A "normal -> generated" replication fails, reporting an error that the
+# subscriber side column is missing.
+#
+# In this test case we use DROP EXPRESSION to change the subscriber gerenated
+# column into a normal column, then verify replication works ok.
+# =============================================================================
+
+# Create publication and table with normal column 'b'
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_alter (a int, b int);
+ CREATE PUBLICATION regress_pub_alter FOR TABLE tab_alter;
+));
+
+# Create subscription and table with a generated column 'b'
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_alter (a int, b int GENERATED ALWAYS AS (a * 22) STORED);
+ CREATE SUBSCRIPTION regress_sub_alter CONNECTION '$publisher_connstr'
+ PUBLICATION regress_pub_alter WITH (copy_data = false);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync;
+
+# Change the generated column 'b' to be a normal column.
+$node_subscriber->safe_psql('postgres',
+ "ALTER TABLE tab_alter ALTER COLUMN b DROP EXPRESSION");
+
+# Insert data to verify replication.
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter VALUES (1,1), (2,2), (3,3)");
+
+# Verify that replication works, now that the subscriber column 'b' is normal
+$node_publisher->wait_for_catchup('regress_sub_alter');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b FROM tab_alter ORDER BY a");
+is($result, qq(1|1
+2|2
+3|3), 'after drop generated column expression');
+
+# cleanup
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION regress_sub_alter");
+$node_publisher->safe_psql('postgres',
+ "DROP PUBLICATION regress_pub_alter");
+
done_testing();
--
1.8.3.1
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-10-09 05:43 Peter Smith <[email protected]>
parent: Shubham Khanna <[email protected]>
1 sibling, 1 reply; 60+ messages in thread
From: Peter Smith @ 2024-10-09 05:43 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi, here are my review comments for patch v37-0001.
======
Commit message
1.
Example usage of subscription option:
CREATE PUBLICATION FOR TABLE tab_gencol WITH (publish_generated_columns
= true);
~
This is wrong -- it's not a "subscription option". Better to just say
"Example usage:"
~~~
2.
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'publish_generated_columns'
is true...
~
By only mentioning the "when ... is true" case this description does
not cover the scenario when 'publish_generated_columns' is false when
the publication column list has a generated column.
~~~
3.
typo - /replication of generated column/replication of generated columns/
typo - /filed/filled/
typo - 'pg_publicataion' catalog
======
src/backend/replication/logical/tablesync.c
make_copy_attnamelist:
4.
nit - missing word in a comment
~~~
fetch_remote_table_info:
5.
+ appendStringInfo(&cmd,
" FROM pg_catalog.pg_attribute a"
" LEFT JOIN pg_catalog.pg_index i"
" ON (i.indexrelid = pg_get_replica_identity_index(%u))"
" WHERE a.attnum > 0::pg_catalog.int2"
- " AND NOT a.attisdropped %s"
+ " AND NOT a.attisdropped", lrel->remoteid);
+
+ appendStringInfo(&cmd,
" AND a.attrelid = %u"
" ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
- "AND a.attgenerated = ''" : ""),
lrel->remoteid);
Version v37-0001 has removed a condition previously between these two
appendStringInfo's. But, that now means there is no reason to keep
these statements separated. These should be combined now to use one
appendStringInfo.
~
6.
+ if (server_version >= 120000)
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+
Are you sure the version check for 120000 is correct? IIUC, this 5
matches the 'attgenerated' column, but the SQL for that was
constructed using a different condition:
if (server_version >= 180000)
appendStringInfo(&cmd, ", a.attgenerated != ''");
It is this 120000 versus 180000 difference that makes me suspicious of
a potential mistake.
~~~
7.
+ /*
+ * If the column is generated and neither the generated column option
+ * is specified nor it appears in the column list, we will skip it.
+ */
+ if (remotegenlist[natt] && !has_pub_with_pubgencols &&
+ !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
7b.
I am also suspicious about how this condition interacts with the other
condition (shown below) that came earlier:
/* If the column is not in the column list, skip it. */
if (included_cols != NULL && !bms_is_member(attnum, included_cols))
Something doesn't seem right. e.g. If we can only get here by passing
the earlier condition, then it means we already know the generated
condition was *not* a member of a column list.... in which case that
should affect this new condition and the new comment too.
======
src/backend/replication/pgoutput/pgoutput.c
pgoutput_column_list_init:
8.
/*
- * If the publication is FOR ALL TABLES then it is treated the same as
- * if there are no column lists (even if other publications have a
- * list).
+ * Process potential column lists for the following cases: a. Any
+ * publication that is not FOR ALL TABLES. b. When the publication is
+ * FOR ALL TABLES and 'publish_generated_columns' is false. FOR ALL
+ * TABLES publication doesn't have user-defined column lists, so all
+ * columns will be replicated by default. However, if
+ * 'publish_generated_columns' is set to false, column lists must
+ * still be created to exclude any generated columns from being
+ * published.
*/
nit - please reformat this comment so the bullets are readable
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6f9e126..365f987 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -693,7 +693,7 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create a list of columns for COPY based on logical relation mapping.
* Exclude columns that are subscription table generated columns.
*/
static List *
@@ -749,7 +749,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
}
/*
- * Construct column list for COPY, excluding columns that are subscription
+ * Construct a column list for COPY, excluding columns that are subscription
* table generated columns.
*/
for (int i = 0; i < rel->remoterel.natts; i++)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d953a1a..6d6032d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1073,13 +1073,15 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
Bitmapset *cols = NULL;
/*
- * Process potential column lists for the following cases: a. Any
- * publication that is not FOR ALL TABLES. b. When the publication is
- * FOR ALL TABLES and 'publish_generated_columns' is false. FOR ALL
- * TABLES publication doesn't have user-defined column lists, so all
- * columns will be replicated by default. However, if
- * 'publish_generated_columns' is set to false, column lists must
- * still be created to exclude any generated columns from being
+ * Process potential column lists for the following cases:
+ *
+ * a. Any publication that is not FOR ALL TABLES.
+ *
+ * b. When the publication is FOR ALL TABLES and
+ * 'publish_generated_columns' is false. FOR ALL TABLES publication doesn't
+ * have user-defined column lists, so all columns will be replicated by
+ * default. However, if 'publish_generated_columns' is set to false, column
+ * lists must still be created to exclude any generated columns from being
* published.
*/
if (!(pub->alltables && pub->pubgencols))
Attachments:
[text/plain] PS_NITPICKS_GENCOLS_v370001.txt (2.2K, ../../CAHut+Pv8Do3b7QhzHg7dRWhO317ZFZKY_mYQaFBOWVQ-P1805A@mail.gmail.com/2-PS_NITPICKS_GENCOLS_v370001.txt)
download | inline diff:
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6f9e126..365f987 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -693,7 +693,7 @@ process_syncing_tables(XLogRecPtr current_lsn)
}
/*
- * Create list of columns for COPY based on logical relation mapping.
+ * Create a list of columns for COPY based on logical relation mapping.
* Exclude columns that are subscription table generated columns.
*/
static List *
@@ -749,7 +749,7 @@ make_copy_attnamelist(LogicalRepRelMapEntry *rel, bool *remotegenlist)
}
/*
- * Construct column list for COPY, excluding columns that are subscription
+ * Construct a column list for COPY, excluding columns that are subscription
* table generated columns.
*/
for (int i = 0; i < rel->remoterel.natts; i++)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d953a1a..6d6032d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1073,13 +1073,15 @@ pgoutput_column_list_init(PGOutputData *data, List *publications,
Bitmapset *cols = NULL;
/*
- * Process potential column lists for the following cases: a. Any
- * publication that is not FOR ALL TABLES. b. When the publication is
- * FOR ALL TABLES and 'publish_generated_columns' is false. FOR ALL
- * TABLES publication doesn't have user-defined column lists, so all
- * columns will be replicated by default. However, if
- * 'publish_generated_columns' is set to false, column lists must
- * still be created to exclude any generated columns from being
+ * Process potential column lists for the following cases:
+ *
+ * a. Any publication that is not FOR ALL TABLES.
+ *
+ * b. When the publication is FOR ALL TABLES and
+ * 'publish_generated_columns' is false. FOR ALL TABLES publication doesn't
+ * have user-defined column lists, so all columns will be replicated by
+ * default. However, if 'publish_generated_columns' is set to false, column
+ * lists must still be created to exclude any generated columns from being
* published.
*/
if (!(pub->alltables && pub->pubgencols))
^ permalink raw reply [nested|flat] 60+ messages in thread
* Re: Pgoutput not capturing the generated columns
@ 2024-10-16 18:01 Shubham Khanna <[email protected]>
parent: Peter Smith <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Shubham Khanna @ 2024-10-16 18:01 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Rajendra Kumar Dangwal <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Oct 9, 2024 at 11:13 AM Peter Smith <[email protected]> wrote:
>
> Hi, here are my review comments for patch v37-0001.
>
> ======
> Commit message
>
> 1.
> Example usage of subscription option:
> CREATE PUBLICATION FOR TABLE tab_gencol WITH (publish_generated_columns
> = true);
>
> ~
>
> This is wrong -- it's not a "subscription option". Better to just say
> "Example usage:"
>
> ~~~
>
> 2.
> When 'copy_data' is true, during the initial sync, the data is replicated from
> the publisher to the subscriber using the COPY command. The normal COPY
> command does not copy generated columns, so when 'publish_generated_columns'
> is true...
>
> ~
>
> By only mentioning the "when ... is true" case this description does
> not cover the scenario when 'publish_generated_columns' is false when
> the publication column list has a generated column.
>
> ~~~
>
> 3.
> typo - /replication of generated column/replication of generated columns/
> typo - /filed/filled/
> typo - 'pg_publicataion' catalog
>
> ======
> src/backend/replication/logical/tablesync.c
>
> make_copy_attnamelist:
> 4.
> nit - missing word in a comment
>
> ~~~
>
> fetch_remote_table_info:
> 5.
> + appendStringInfo(&cmd,
> " FROM pg_catalog.pg_attribute a"
> " LEFT JOIN pg_catalog.pg_index i"
> " ON (i.indexrelid = pg_get_replica_identity_index(%u))"
> " WHERE a.attnum > 0::pg_catalog.int2"
> - " AND NOT a.attisdropped %s"
> + " AND NOT a.attisdropped", lrel->remoteid);
> +
> + appendStringInfo(&cmd,
> " AND a.attrelid = %u"
> " ORDER BY a.attnum",
> - lrel->remoteid,
> - (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
> - "AND a.attgenerated = ''" : ""),
> lrel->remoteid);
>
> Version v37-0001 has removed a condition previously between these two
> appendStringInfo's. But, that now means there is no reason to keep
> these statements separated. These should be combined now to use one
> appendStringInfo.
>
> ~
>
> 6.
> + if (server_version >= 120000)
> + remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
> +
>
> Are you sure the version check for 120000 is correct? IIUC, this 5
> matches the 'attgenerated' column, but the SQL for that was
> constructed using a different condition:
> if (server_version >= 180000)
> appendStringInfo(&cmd, ", a.attgenerated != ''");
>
> It is this 120000 versus 180000 difference that makes me suspicious of
> a potential mistake.
>
> ~~~
>
> 7.
> + /*
> + * If the column is generated and neither the generated column option
> + * is specified nor it appears in the column list, we will skip it.
> + */
> + if (remotegenlist[natt] && !has_pub_with_pubgencols &&
> + !bms_is_member(attnum, included_cols))
> + {
> + ExecClearTuple(slot);
> + continue;
> + }
>
> 7b.
> I am also suspicious about how this condition interacts with the other
> condition (shown below) that came earlier:
> /* If the column is not in the column list, skip it. */
> if (included_cols != NULL && !bms_is_member(attnum, included_cols))
>
> Something doesn't seem right. e.g. If we can only get here by passing
> the earlier condition, then it means we already know the generated
> condition was *not* a member of a column list.... in which case that
> should affect this new condition and the new comment too.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> pgoutput_column_list_init:
>
> 8.
> /*
> - * If the publication is FOR ALL TABLES then it is treated the same as
> - * if there are no column lists (even if other publications have a
> - * list).
> + * Process potential column lists for the following cases: a. Any
> + * publication that is not FOR ALL TABLES. b. When the publication is
> + * FOR ALL TABLES and 'publish_generated_columns' is false. FOR ALL
> + * TABLES publication doesn't have user-defined column lists, so all
> + * columns will be replicated by default. However, if
> + * 'publish_generated_columns' is set to false, column lists must
> + * still be created to exclude any generated columns from being
> + * published.
> */
>
> nit - please reformat this comment so the bullets are readable
>
I have fixed all the comments and posted the v39 patches for them.
Please refer to the updated v39 Patches here in [1]. See [1] for the
changes added.
[1] https://www.postgresql.org/message-id/CAHv8RjLjb%2B98i5ZQUphivxdOZ3hSGLfq2SiWQetUvk8zGyAQwQ%40mail.g...
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 60+ messages in thread
* [PATCH v1 1/1] remove VersionedQuery support from psql tab completion
@ 2026-07-02 18:39 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Nathan Bossart @ 2026-07-02 18:39 UTC (permalink / raw)
---
src/bin/psql/tab-complete.in.c | 150 ++++++++++-----------------------
1 file changed, 45 insertions(+), 105 deletions(-)
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index e4722c06664..c238d466e29 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -92,24 +92,6 @@
*/
PQExpBuffer tab_completion_query_buf = NULL;
-/*
- * In some situations, the query to find out what names are available to
- * complete with must vary depending on server version. We handle this by
- * storing a list of queries, each tagged with the minimum server version
- * it will work for. Each list must be stored in descending server version
- * order, so that the first satisfactory query is the one to use.
- *
- * When the query string is otherwise constant, an array of VersionedQuery
- * suffices. Terminate the array with an entry having min_server_version = 0.
- * That entry's query string can be a query that works in all supported older
- * server versions, or NULL to give up and do no completion.
- */
-typedef struct VersionedQuery
-{
- int min_server_version;
- const char *query;
-} VersionedQuery;
-
/*
* This struct is used to define "schema queries", which are custom-built
* to obtain possibly-schema-qualified names of database objects. There is
@@ -123,8 +105,7 @@ typedef struct VersionedQuery
* objects we're completing might not have a schema of their own, but the
* reference object almost always does (passed in completion_ref_schema).
*
- * As with VersionedQuery, we can use an array of these if the query details
- * must vary across versions.
+ * We can use an array of these if the query details must vary across versions.
*/
typedef struct SchemaQuery
{
@@ -225,7 +206,6 @@ static int completion_max_records;
static char completion_last_char; /* last char of input word */
static const char *completion_charp; /* to pass a string */
static const char *const *completion_charpp; /* to pass a list of strings */
-static const VersionedQuery *completion_vquery; /* to pass a VersionedQuery */
static const SchemaQuery *completion_squery; /* to pass a SchemaQuery */
static char *completion_ref_object; /* name of reference object */
static char *completion_ref_schema; /* schema name of reference object */
@@ -237,7 +217,6 @@ static bool completion_force_quote; /* true to force-quote filenames */
* A few macros to ease typing. You can use these to complete the given
* string with
* 1) The result from a query you pass it. (Perhaps one of those below?)
- * We support both simple and versioned queries.
* 2) The result from a schema query you pass it.
* We support both simple and versioned schema queries.
* 3) The items from a null-pointer-terminated list (with or without
@@ -288,23 +267,6 @@ do { \
COMPLETE_WITH_QUERY_VERBATIM_LIST(query, list); \
} while (0)
-#define COMPLETE_WITH_VERSIONED_QUERY(query) \
- COMPLETE_WITH_VERSIONED_QUERY_LIST(query, NULL)
-
-#define COMPLETE_WITH_VERSIONED_QUERY_LIST(query, list) \
-do { \
- completion_vquery = query; \
- completion_charpp = list; \
- completion_verbatim = false; \
- matches = rl_completion_matches(text, complete_from_versioned_query); \
-} while (0)
-
-#define COMPLETE_WITH_VERSIONED_QUERY_PLUS(query, ...) \
-do { \
- static const char *const list[] = { __VA_ARGS__, NULL }; \
- COMPLETE_WITH_VERSIONED_QUERY_LIST(query, list); \
-} while (0)
-
#define COMPLETE_WITH_SCHEMA_QUERY(query) \
COMPLETE_WITH_SCHEMA_QUERY_LIST(query, NULL)
@@ -1278,9 +1240,8 @@ static const char *const sql_commands[] = {
typedef struct
{
const char *name;
- /* Provide at most one of these three types of query: */
+ /* Provide at most one of these two types of query: */
const char *query; /* simple query, or NULL */
- const VersionedQuery *vquery; /* versioned query, or NULL */
const SchemaQuery *squery; /* schema query, or NULL */
const char *const *keywords; /* keywords to be offered as well */
const uint32 flags; /* visibility flags, see below */
@@ -1298,68 +1259,68 @@ static const char *const Keywords_for_user_thing[] = {
};
static const pgsql_thing_t words_after_create[] = {
- {"ACCESS METHOD", NULL, NULL, NULL, NULL, THING_NO_ALTER},
- {"AGGREGATE", NULL, NULL, Query_for_list_of_aggregates},
- {"CAST", NULL, NULL, NULL}, /* Casts have complex structures for names, so
+ {"ACCESS METHOD", NULL, NULL, NULL, THING_NO_ALTER},
+ {"AGGREGATE", NULL, Query_for_list_of_aggregates},
+ {"CAST", NULL, NULL}, /* Casts have complex structures for names, so
* skip it */
- {"COLLATION", NULL, NULL, &Query_for_list_of_collations},
+ {"COLLATION", NULL, &Query_for_list_of_collations},
/*
* CREATE CONSTRAINT TRIGGER is not supported here because it is designed
* to be used only by pg_dump.
*/
- {"CONFIGURATION", NULL, NULL, &Query_for_list_of_ts_configurations, NULL, THING_NO_SHOW},
+ {"CONFIGURATION", NULL, &Query_for_list_of_ts_configurations, NULL, THING_NO_SHOW},
{"CONVERSION", "SELECT conname FROM pg_catalog.pg_conversion WHERE conname LIKE '%s'"},
{"DATABASE", Query_for_list_of_databases},
- {"DEFAULT PRIVILEGES", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
- {"DICTIONARY", NULL, NULL, &Query_for_list_of_ts_dictionaries, NULL, THING_NO_SHOW},
- {"DOMAIN", NULL, NULL, &Query_for_list_of_domains},
- {"EVENT TRIGGER", NULL, NULL, NULL},
+ {"DEFAULT PRIVILEGES", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
+ {"DICTIONARY", NULL, &Query_for_list_of_ts_dictionaries, NULL, THING_NO_SHOW},
+ {"DOMAIN", NULL, &Query_for_list_of_domains},
+ {"EVENT TRIGGER", NULL, NULL},
{"EXTENSION", Query_for_list_of_extensions},
- {"FOREIGN DATA WRAPPER", NULL, NULL, NULL},
- {"FOREIGN TABLE", NULL, NULL, NULL},
- {"FUNCTION", NULL, NULL, Query_for_list_of_functions},
+ {"FOREIGN DATA WRAPPER", NULL, NULL},
+ {"FOREIGN TABLE", NULL, NULL},
+ {"FUNCTION", NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
- {"INDEX", NULL, NULL, &Query_for_list_of_indexes},
+ {"INDEX", NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
- {"LARGE OBJECT", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
- {"MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
- {"OPERATOR", NULL, NULL, NULL}, /* Querying for this is probably not such
- * a good idea. */
- {"OR REPLACE", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER},
- {"OWNED", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_ALTER}, /* for DROP OWNED BY ... */
- {"PARSER", NULL, NULL, &Query_for_list_of_ts_parsers, NULL, THING_NO_SHOW},
- {"POLICY", NULL, NULL, NULL},
- {"PROCEDURE", NULL, NULL, Query_for_list_of_procedures},
- {"PROPERTY GRAPH", NULL, NULL, &Query_for_list_of_propgraphs},
+ {"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
+ {"MATERIALIZED VIEW", NULL, &Query_for_list_of_matviews},
+ {"OPERATOR", NULL, NULL}, /* Querying for this is probably not such a
+ * good idea. */
+ {"OR REPLACE", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER},
+ {"OWNED", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_ALTER}, /* for DROP OWNED BY ... */
+ {"PARSER", NULL, &Query_for_list_of_ts_parsers, NULL, THING_NO_SHOW},
+ {"POLICY", NULL, NULL},
+ {"PROCEDURE", NULL, Query_for_list_of_procedures},
+ {"PROPERTY GRAPH", NULL, &Query_for_list_of_propgraphs},
{"PUBLICATION", Query_for_list_of_publications},
{"ROLE", Query_for_list_of_roles},
- {"ROUTINE", NULL, NULL, &Query_for_list_of_routines, NULL, THING_NO_CREATE},
+ {"ROUTINE", NULL, &Query_for_list_of_routines, NULL, THING_NO_CREATE},
{"RULE", "SELECT rulename FROM pg_catalog.pg_rules WHERE rulename LIKE '%s'"},
{"SCHEMA", Query_for_list_of_schemas},
- {"SEQUENCE", NULL, NULL, &Query_for_list_of_sequences},
+ {"SEQUENCE", NULL, &Query_for_list_of_sequences},
{"SERVER", Query_for_list_of_servers},
- {"STATISTICS", NULL, NULL, &Query_for_list_of_statistics},
+ {"STATISTICS", NULL, &Query_for_list_of_statistics},
{"SUBSCRIPTION", Query_for_list_of_subscriptions},
- {"SYSTEM", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
- {"TABLE", NULL, NULL, &Query_for_list_of_tables},
+ {"SYSTEM", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
+ {"TABLE", NULL, &Query_for_list_of_tables},
{"TABLESPACE", Query_for_list_of_tablespaces},
- {"TEMP", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMP TABLE
- * ... */
- {"TEMPLATE", NULL, NULL, &Query_for_list_of_ts_templates, NULL, THING_NO_SHOW},
- {"TEMPORARY", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMPORARY
- * TABLE ... */
- {"TEXT SEARCH", NULL, NULL, NULL},
- {"TRANSFORM", NULL, NULL, NULL, NULL, THING_NO_ALTER},
+ {"TEMP", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMP TABLE
+ * ... */
+ {"TEMPLATE", NULL, &Query_for_list_of_ts_templates, NULL, THING_NO_SHOW},
+ {"TEMPORARY", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMPORARY
+ * TABLE ... */
+ {"TEXT SEARCH", NULL, NULL},
+ {"TRANSFORM", NULL, NULL, NULL, THING_NO_ALTER},
{"TRIGGER", "SELECT tgname FROM pg_catalog.pg_trigger WHERE tgname LIKE '%s' AND NOT tgisinternal"},
- {"TYPE", NULL, NULL, &Query_for_list_of_datatypes},
- {"UNIQUE", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNIQUE
- * INDEX ... */
- {"UNLOGGED", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNLOGGED
- * TABLE ... */
- {"USER", Query_for_list_of_roles, NULL, NULL, Keywords_for_user_thing},
- {"USER MAPPING FOR", NULL, NULL, NULL},
- {"VIEW", NULL, NULL, &Query_for_list_of_views},
+ {"TYPE", NULL, &Query_for_list_of_datatypes},
+ {"UNIQUE", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNIQUE
+ * INDEX ... */
+ {"UNLOGGED", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNLOGGED
+ * TABLE ... */
+ {"USER", Query_for_list_of_roles, NULL, Keywords_for_user_thing},
+ {"USER MAPPING FOR", NULL, NULL},
+ {"VIEW", NULL, &Query_for_list_of_views},
{NULL} /* end of list */
};
@@ -1475,7 +1436,6 @@ static char *create_command_generator(const char *text, int state);
static char *drop_command_generator(const char *text, int state);
static char *alter_command_generator(const char *text, int state);
static char *complete_from_query(const char *text, int state);
-static char *complete_from_versioned_query(const char *text, int state);
static char *complete_from_schema_query(const char *text, int state);
static char *complete_from_versioned_schema_query(const char *text, int state);
static char *_complete_from_query(const char *simple_query,
@@ -1967,7 +1927,6 @@ psql_completion(const char *text, int start, int end)
/* Clear a few things. */
completion_charp = NULL;
completion_charpp = NULL;
- completion_vquery = NULL;
completion_squery = NULL;
completion_ref_object = NULL;
completion_ref_schema = NULL;
@@ -2098,9 +2057,6 @@ psql_completion(const char *text, int start, int end)
if (wac->query)
COMPLETE_WITH_QUERY_LIST(wac->query,
wac->keywords);
- else if (wac->vquery)
- COMPLETE_WITH_VERSIONED_QUERY_LIST(wac->vquery,
- wac->keywords);
else if (wac->squery)
COMPLETE_WITH_VERSIONED_SCHEMA_QUERY_LIST(wac->squery,
wac->keywords);
@@ -5928,22 +5884,6 @@ complete_from_query(const char *text, int state)
completion_verbatim, text, state);
}
-static char *
-complete_from_versioned_query(const char *text, int state)
-{
- const VersionedQuery *vquery = completion_vquery;
-
- /* Find appropriate array element */
- while (pset.sversion < vquery->min_server_version)
- vquery++;
- /* Fail completion if server is too old */
- if (vquery->query == NULL)
- return NULL;
-
- return _complete_from_query(vquery->query, NULL, completion_charpp,
- completion_verbatim, text, state);
-}
-
static char *
complete_from_schema_query(const char *text, int state)
{
--
2.50.1 (Apple Git-155)
--m8LJOl54+zCjPq94--
^ permalink raw reply [nested|flat] 60+ messages in thread
* [PATCH v1 1/1] remove VersionedQuery support from psql tab completion
@ 2026-07-02 18:39 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 60+ messages in thread
From: Nathan Bossart @ 2026-07-02 18:39 UTC (permalink / raw)
---
src/bin/psql/tab-complete.in.c | 150 ++++++++++-----------------------
1 file changed, 45 insertions(+), 105 deletions(-)
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index e4722c06664..c238d466e29 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -92,24 +92,6 @@
*/
PQExpBuffer tab_completion_query_buf = NULL;
-/*
- * In some situations, the query to find out what names are available to
- * complete with must vary depending on server version. We handle this by
- * storing a list of queries, each tagged with the minimum server version
- * it will work for. Each list must be stored in descending server version
- * order, so that the first satisfactory query is the one to use.
- *
- * When the query string is otherwise constant, an array of VersionedQuery
- * suffices. Terminate the array with an entry having min_server_version = 0.
- * That entry's query string can be a query that works in all supported older
- * server versions, or NULL to give up and do no completion.
- */
-typedef struct VersionedQuery
-{
- int min_server_version;
- const char *query;
-} VersionedQuery;
-
/*
* This struct is used to define "schema queries", which are custom-built
* to obtain possibly-schema-qualified names of database objects. There is
@@ -123,8 +105,7 @@ typedef struct VersionedQuery
* objects we're completing might not have a schema of their own, but the
* reference object almost always does (passed in completion_ref_schema).
*
- * As with VersionedQuery, we can use an array of these if the query details
- * must vary across versions.
+ * We can use an array of these if the query details must vary across versions.
*/
typedef struct SchemaQuery
{
@@ -225,7 +206,6 @@ static int completion_max_records;
static char completion_last_char; /* last char of input word */
static const char *completion_charp; /* to pass a string */
static const char *const *completion_charpp; /* to pass a list of strings */
-static const VersionedQuery *completion_vquery; /* to pass a VersionedQuery */
static const SchemaQuery *completion_squery; /* to pass a SchemaQuery */
static char *completion_ref_object; /* name of reference object */
static char *completion_ref_schema; /* schema name of reference object */
@@ -237,7 +217,6 @@ static bool completion_force_quote; /* true to force-quote filenames */
* A few macros to ease typing. You can use these to complete the given
* string with
* 1) The result from a query you pass it. (Perhaps one of those below?)
- * We support both simple and versioned queries.
* 2) The result from a schema query you pass it.
* We support both simple and versioned schema queries.
* 3) The items from a null-pointer-terminated list (with or without
@@ -288,23 +267,6 @@ do { \
COMPLETE_WITH_QUERY_VERBATIM_LIST(query, list); \
} while (0)
-#define COMPLETE_WITH_VERSIONED_QUERY(query) \
- COMPLETE_WITH_VERSIONED_QUERY_LIST(query, NULL)
-
-#define COMPLETE_WITH_VERSIONED_QUERY_LIST(query, list) \
-do { \
- completion_vquery = query; \
- completion_charpp = list; \
- completion_verbatim = false; \
- matches = rl_completion_matches(text, complete_from_versioned_query); \
-} while (0)
-
-#define COMPLETE_WITH_VERSIONED_QUERY_PLUS(query, ...) \
-do { \
- static const char *const list[] = { __VA_ARGS__, NULL }; \
- COMPLETE_WITH_VERSIONED_QUERY_LIST(query, list); \
-} while (0)
-
#define COMPLETE_WITH_SCHEMA_QUERY(query) \
COMPLETE_WITH_SCHEMA_QUERY_LIST(query, NULL)
@@ -1278,9 +1240,8 @@ static const char *const sql_commands[] = {
typedef struct
{
const char *name;
- /* Provide at most one of these three types of query: */
+ /* Provide at most one of these two types of query: */
const char *query; /* simple query, or NULL */
- const VersionedQuery *vquery; /* versioned query, or NULL */
const SchemaQuery *squery; /* schema query, or NULL */
const char *const *keywords; /* keywords to be offered as well */
const uint32 flags; /* visibility flags, see below */
@@ -1298,68 +1259,68 @@ static const char *const Keywords_for_user_thing[] = {
};
static const pgsql_thing_t words_after_create[] = {
- {"ACCESS METHOD", NULL, NULL, NULL, NULL, THING_NO_ALTER},
- {"AGGREGATE", NULL, NULL, Query_for_list_of_aggregates},
- {"CAST", NULL, NULL, NULL}, /* Casts have complex structures for names, so
+ {"ACCESS METHOD", NULL, NULL, NULL, THING_NO_ALTER},
+ {"AGGREGATE", NULL, Query_for_list_of_aggregates},
+ {"CAST", NULL, NULL}, /* Casts have complex structures for names, so
* skip it */
- {"COLLATION", NULL, NULL, &Query_for_list_of_collations},
+ {"COLLATION", NULL, &Query_for_list_of_collations},
/*
* CREATE CONSTRAINT TRIGGER is not supported here because it is designed
* to be used only by pg_dump.
*/
- {"CONFIGURATION", NULL, NULL, &Query_for_list_of_ts_configurations, NULL, THING_NO_SHOW},
+ {"CONFIGURATION", NULL, &Query_for_list_of_ts_configurations, NULL, THING_NO_SHOW},
{"CONVERSION", "SELECT conname FROM pg_catalog.pg_conversion WHERE conname LIKE '%s'"},
{"DATABASE", Query_for_list_of_databases},
- {"DEFAULT PRIVILEGES", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
- {"DICTIONARY", NULL, NULL, &Query_for_list_of_ts_dictionaries, NULL, THING_NO_SHOW},
- {"DOMAIN", NULL, NULL, &Query_for_list_of_domains},
- {"EVENT TRIGGER", NULL, NULL, NULL},
+ {"DEFAULT PRIVILEGES", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
+ {"DICTIONARY", NULL, &Query_for_list_of_ts_dictionaries, NULL, THING_NO_SHOW},
+ {"DOMAIN", NULL, &Query_for_list_of_domains},
+ {"EVENT TRIGGER", NULL, NULL},
{"EXTENSION", Query_for_list_of_extensions},
- {"FOREIGN DATA WRAPPER", NULL, NULL, NULL},
- {"FOREIGN TABLE", NULL, NULL, NULL},
- {"FUNCTION", NULL, NULL, Query_for_list_of_functions},
+ {"FOREIGN DATA WRAPPER", NULL, NULL},
+ {"FOREIGN TABLE", NULL, NULL},
+ {"FUNCTION", NULL, Query_for_list_of_functions},
{"GROUP", Query_for_list_of_roles},
- {"INDEX", NULL, NULL, &Query_for_list_of_indexes},
+ {"INDEX", NULL, &Query_for_list_of_indexes},
{"LANGUAGE", Query_for_list_of_languages},
- {"LARGE OBJECT", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
- {"MATERIALIZED VIEW", NULL, NULL, &Query_for_list_of_matviews},
- {"OPERATOR", NULL, NULL, NULL}, /* Querying for this is probably not such
- * a good idea. */
- {"OR REPLACE", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER},
- {"OWNED", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_ALTER}, /* for DROP OWNED BY ... */
- {"PARSER", NULL, NULL, &Query_for_list_of_ts_parsers, NULL, THING_NO_SHOW},
- {"POLICY", NULL, NULL, NULL},
- {"PROCEDURE", NULL, NULL, Query_for_list_of_procedures},
- {"PROPERTY GRAPH", NULL, NULL, &Query_for_list_of_propgraphs},
+ {"LARGE OBJECT", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
+ {"MATERIALIZED VIEW", NULL, &Query_for_list_of_matviews},
+ {"OPERATOR", NULL, NULL}, /* Querying for this is probably not such a
+ * good idea. */
+ {"OR REPLACE", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER},
+ {"OWNED", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_ALTER}, /* for DROP OWNED BY ... */
+ {"PARSER", NULL, &Query_for_list_of_ts_parsers, NULL, THING_NO_SHOW},
+ {"POLICY", NULL, NULL},
+ {"PROCEDURE", NULL, Query_for_list_of_procedures},
+ {"PROPERTY GRAPH", NULL, &Query_for_list_of_propgraphs},
{"PUBLICATION", Query_for_list_of_publications},
{"ROLE", Query_for_list_of_roles},
- {"ROUTINE", NULL, NULL, &Query_for_list_of_routines, NULL, THING_NO_CREATE},
+ {"ROUTINE", NULL, &Query_for_list_of_routines, NULL, THING_NO_CREATE},
{"RULE", "SELECT rulename FROM pg_catalog.pg_rules WHERE rulename LIKE '%s'"},
{"SCHEMA", Query_for_list_of_schemas},
- {"SEQUENCE", NULL, NULL, &Query_for_list_of_sequences},
+ {"SEQUENCE", NULL, &Query_for_list_of_sequences},
{"SERVER", Query_for_list_of_servers},
- {"STATISTICS", NULL, NULL, &Query_for_list_of_statistics},
+ {"STATISTICS", NULL, &Query_for_list_of_statistics},
{"SUBSCRIPTION", Query_for_list_of_subscriptions},
- {"SYSTEM", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
- {"TABLE", NULL, NULL, &Query_for_list_of_tables},
+ {"SYSTEM", NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP},
+ {"TABLE", NULL, &Query_for_list_of_tables},
{"TABLESPACE", Query_for_list_of_tablespaces},
- {"TEMP", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMP TABLE
- * ... */
- {"TEMPLATE", NULL, NULL, &Query_for_list_of_ts_templates, NULL, THING_NO_SHOW},
- {"TEMPORARY", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMPORARY
- * TABLE ... */
- {"TEXT SEARCH", NULL, NULL, NULL},
- {"TRANSFORM", NULL, NULL, NULL, NULL, THING_NO_ALTER},
+ {"TEMP", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMP TABLE
+ * ... */
+ {"TEMPLATE", NULL, &Query_for_list_of_ts_templates, NULL, THING_NO_SHOW},
+ {"TEMPORARY", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE TEMPORARY
+ * TABLE ... */
+ {"TEXT SEARCH", NULL, NULL},
+ {"TRANSFORM", NULL, NULL, NULL, THING_NO_ALTER},
{"TRIGGER", "SELECT tgname FROM pg_catalog.pg_trigger WHERE tgname LIKE '%s' AND NOT tgisinternal"},
- {"TYPE", NULL, NULL, &Query_for_list_of_datatypes},
- {"UNIQUE", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNIQUE
- * INDEX ... */
- {"UNLOGGED", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNLOGGED
- * TABLE ... */
- {"USER", Query_for_list_of_roles, NULL, NULL, Keywords_for_user_thing},
- {"USER MAPPING FOR", NULL, NULL, NULL},
- {"VIEW", NULL, NULL, &Query_for_list_of_views},
+ {"TYPE", NULL, &Query_for_list_of_datatypes},
+ {"UNIQUE", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNIQUE
+ * INDEX ... */
+ {"UNLOGGED", NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE UNLOGGED
+ * TABLE ... */
+ {"USER", Query_for_list_of_roles, NULL, Keywords_for_user_thing},
+ {"USER MAPPING FOR", NULL, NULL},
+ {"VIEW", NULL, &Query_for_list_of_views},
{NULL} /* end of list */
};
@@ -1475,7 +1436,6 @@ static char *create_command_generator(const char *text, int state);
static char *drop_command_generator(const char *text, int state);
static char *alter_command_generator(const char *text, int state);
static char *complete_from_query(const char *text, int state);
-static char *complete_from_versioned_query(const char *text, int state);
static char *complete_from_schema_query(const char *text, int state);
static char *complete_from_versioned_schema_query(const char *text, int state);
static char *_complete_from_query(const char *simple_query,
@@ -1967,7 +1927,6 @@ psql_completion(const char *text, int start, int end)
/* Clear a few things. */
completion_charp = NULL;
completion_charpp = NULL;
- completion_vquery = NULL;
completion_squery = NULL;
completion_ref_object = NULL;
completion_ref_schema = NULL;
@@ -2098,9 +2057,6 @@ psql_completion(const char *text, int start, int end)
if (wac->query)
COMPLETE_WITH_QUERY_LIST(wac->query,
wac->keywords);
- else if (wac->vquery)
- COMPLETE_WITH_VERSIONED_QUERY_LIST(wac->vquery,
- wac->keywords);
else if (wac->squery)
COMPLETE_WITH_VERSIONED_SCHEMA_QUERY_LIST(wac->squery,
wac->keywords);
@@ -5928,22 +5884,6 @@ complete_from_query(const char *text, int state)
completion_verbatim, text, state);
}
-static char *
-complete_from_versioned_query(const char *text, int state)
-{
- const VersionedQuery *vquery = completion_vquery;
-
- /* Find appropriate array element */
- while (pset.sversion < vquery->min_server_version)
- vquery++;
- /* Fail completion if server is too old */
- if (vquery->query == NULL)
- return NULL;
-
- return _complete_from_query(vquery->query, NULL, completion_charpp,
- completion_verbatim, text, state);
-}
-
static char *
complete_from_schema_query(const char *text, int state)
{
--
2.50.1 (Apple Git-155)
--m8LJOl54+zCjPq94--
^ permalink raw reply [nested|flat] 60+ messages in thread
end of thread, other threads:[~2026-07-02 18:39 UTC | newest]
Thread overview: 60+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-06-20 03:33 Re: Pgoutput not capturing the generated columns Peter Smith <[email protected]>
2024-06-21 02:52 ` Peter Smith <[email protected]>
2024-06-22 18:08 ` Shubham Khanna <[email protected]>
2024-06-22 18:06 ` Shubham Khanna <[email protected]>
2024-06-23 04:58 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-06-26 12:12 ` Shubham Khanna <[email protected]>
2024-06-27 05:29 ` Shubham Khanna <[email protected]>
2024-06-27 09:10 ` Peter Smith <[email protected]>
2024-07-01 10:38 ` Shubham Khanna <[email protected]>
2024-07-02 04:23 ` Peter Smith <[email protected]>
2024-07-05 12:01 ` Shubham Khanna <[email protected]>
2024-07-08 05:22 ` Peter Smith <[email protected]>
2024-07-11 06:12 ` Shubham Khanna <[email protected]>
2024-07-12 06:43 ` Peter Smith <[email protected]>
2024-07-16 11:19 ` Shubham Khanna <[email protected]>
2024-07-18 05:17 ` Peter Smith <[email protected]>
2024-07-26 12:23 ` Shubham Khanna <[email protected]>
2024-07-29 07:26 ` Peter Smith <[email protected]>
2024-07-31 08:42 ` Shubham Khanna <[email protected]>
2024-08-01 08:32 ` Peter Smith <[email protected]>
2024-08-02 04:26 ` Shubham Khanna <[email protected]>
2024-08-05 02:40 ` Peter Smith <[email protected]>
2024-08-05 03:44 ` Peter Smith <[email protected]>
2024-08-07 05:37 ` Shubham Khanna <[email protected]>
2024-08-07 05:26 ` Shubham Khanna <[email protected]>
2024-08-07 08:00 ` Peter Smith <[email protected]>
2024-08-08 05:23 ` Shubham Khanna <[email protected]>
2024-08-08 07:13 ` Peter Smith <[email protected]>
2024-08-16 04:34 ` Shubham Khanna <[email protected]>
2024-08-16 09:17 ` vignesh C <[email protected]>
2024-08-22 04:52 ` Shubham Khanna <[email protected]>
2024-08-22 11:58 ` vignesh C <[email protected]>
2024-08-23 10:56 ` Peter Smith <[email protected]>
2024-10-09 05:43 ` Peter Smith <[email protected]>
2024-10-16 18:01 ` Shubham Khanna <[email protected]>
2024-08-10 14:23 ` vignesh C <[email protected]>
2024-08-16 11:10 ` Shubham Khanna <[email protected]>
2024-08-19 05:31 ` Peter Smith <[email protected]>
2024-08-22 04:56 ` Shubham Khanna <[email protected]>
2024-08-19 07:10 ` Peter Smith <[email protected]>
2024-08-22 04:58 ` Shubham Khanna <[email protected]>
2024-07-18 08:24 ` Peter Smith <[email protected]>
2024-07-19 06:01 ` Shlok Kyal <[email protected]>
2024-07-22 23:23 ` Peter Smith <[email protected]>
2024-08-15 07:39 ` Peter Smith <[email protected]>
2024-07-18 23:29 ` Peter Smith <[email protected]>
2024-07-19 06:03 ` Shlok Kyal <[email protected]>
2024-07-15 02:38 ` Peter Smith <[email protected]>
2024-07-16 09:04 ` Shlok Kyal <[email protected]>
2024-07-15 05:39 ` Peter Smith <[email protected]>
2024-07-08 07:50 ` Peter Smith <[email protected]>
2024-07-08 12:04 ` Shlok Kyal <[email protected]>
2024-07-09 04:22 ` Peter Smith <[email protected]>
2024-07-16 08:58 ` Shlok Kyal <[email protected]>
2024-07-02 05:29 ` Peter Smith <[email protected]>
2024-07-05 12:05 ` Shubham Khanna <[email protected]>
2024-06-24 02:51 ` Peter Smith <[email protected]>
2024-06-26 12:16 ` Shubham Khanna <[email protected]>
2026-07-02 18:39 [PATCH v1 1/1] remove VersionedQuery support from psql tab completion Nathan Bossart <[email protected]>
2026-07-02 18:39 [PATCH v1 1/1] remove VersionedQuery support from psql tab completion Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox