public inbox for [email protected]
help / color / mirror / Atom feedRe: RFC: logical publication via inheritance root?
18+ messages / 5 participants
[nested] [flat]
* Re: RFC: logical publication via inheritance root?
@ 2023-01-06 21:55 Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2023-01-06 21:55 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
On Fri, Dec 9, 2022 at 10:21 AM Jacob Champion <[email protected]> wrote:
> Some inheritance hierarchies won't be "partitioned" hierarchies, of
> course, but the user can simply not set that replication option for
> those publications.
The more I noodle around with this approach, the less I like it: it
feels overly brittle, we have to deal with multiple inheritance
somehow, and there seem to be many code paths that need to be
partially duplicated. And my suggestion that the user could just opt
out of problematic cases would be a bad user experience, since any
non-partition inheritance hierarchies would just silently break.
Instead...
> (Alternatively, I can imagine a system where an
> extension explicitly marks a table as having a different "publication
> root", and then handling that marker with the existing replication
> option. But that may be overengineering things.)
...I'm going to try this approach next, since it's opt-in and may be
able to better use the existing code paths.
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-01-09 08:41 ` Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Aleksander Alekseev @ 2023-01-09 08:41 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Jacob,
> we have to deal with multiple inheritance somehow
I would like to point out that we shouldn't necessarily support
multiple inheritance in all the possible cases, at least not in the
first implementation. Supporting simple cases of inheritance would be
already a valuable feature even if it will have certain limitations.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
@ 2023-01-10 19:36 ` Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2023-01-10 19:36 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Mon, Jan 9, 2023 at 12:41 AM Aleksander Alekseev
<[email protected]> wrote:
> I would like to point out that we shouldn't necessarily support
> multiple inheritance in all the possible cases, at least not in the
> first implementation. Supporting simple cases of inheritance would be
> already a valuable feature even if it will have certain limitations.
I agree. What I'm trying to avoid is the case where replication works
nicely for a table until someone performs an ALTER TABLE ... [NO]
INHERIT, and then Something Bad happens because we can't support the
new edge case. If every inheritance tree is automatically opted into
this new publication behavior, I think it'll be easier to hit that by
accident, making the whole thing feel brittle.
By contrast, if we have to opt specific tables into this feature by
marking them in the catalog, then not only will it be harder to hit by
accident (because we can document the requirements for the marker
function, and then it's up to the callers/extension authors/DBAs to
maintain those requirements), but we even have the chance to bail out
during an inheritance change if we see that the table is marked in
this way.
Two general pieces of progress to report:
1) I'm playing around with a marker in pg_inherits, where the inhseqno
is set to a sentinel value (0) for an inheritance relationship that
has been marked for logical publication. The intent is that the
pg_inherits helpers will prevent further inheritance relationships
when they see that marker, and reusing inhseqno means we can make use
of the existing index to do the lookups. An example:
=# CREATE TABLE root (a int);
=# CREATE TABLE root_p1 () INHERITS (root);
=# SELECT pg_set_logical_root('root_p1', 'root');
and then any data written to root_p1 gets replicated via root instead,
if publish_via_partition_root = true. If root_p1 is set up with extra
columns, they'll be omitted from replication.
2) While this strategy works well for ongoing replication, it's not
enough to get the initial synchronization correct. The subscriber
still does a COPY of the root table directly, missing out on all the
logical descendant data. The publisher will have to tell the
subscriber about the relationship somehow, and older subscriber
versions won't understand how to use that (similar to how old
subscribers can't correctly handle row filters).
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-01-20 17:53 ` Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2023-01-20 17:53 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 1/10/23 11:36, Jacob Champion wrote:
> 1) I'm playing around with a marker in pg_inherits, where the inhseqno
> is set to a sentinel value (0) for an inheritance relationship that
> has been marked for logical publication. The intent is that the
> pg_inherits helpers will prevent further inheritance relationships
> when they see that marker, and reusing inhseqno means we can make use
> of the existing index to do the lookups. An example:
>
> =# CREATE TABLE root (a int);
> =# CREATE TABLE root_p1 () INHERITS (root);
> =# SELECT pg_set_logical_root('root_p1', 'root');
>
> and then any data written to root_p1 gets replicated via root instead,
> if publish_via_partition_root = true. If root_p1 is set up with extra
> columns, they'll be omitted from replication.
First draft attached. (Due to some indentation changes, it's easiest to
read with --ignore-all-space.)
The overall strategy is
- introduce pg_set_logical_root, which sets the sentinel in pg_inherits,
- swap out any checks for partition parents with checks for logical
parents in the publishing code, and
- introduce the ability for a subscriber to perform an initial table
sync from multiple tables on the publisher.
> 2) While this strategy works well for ongoing replication, it's not
> enough to get the initial synchronization correct. The subscriber
> still does a COPY of the root table directly, missing out on all the
> logical descendant data. The publisher will have to tell the
> subscriber about the relationship somehow, and older subscriber
> versions won't understand how to use that (similar to how old
> subscribers can't correctly handle row filters).
I partially solved this by having the subscriber pull the logical
hierarchy from the publisher to figure out which tables to COPY. This
works when publish_via_partition_root=true, but it doesn't correctly
return to the previous behavior when the setting is false. I need to
check the publication setting from the subscriber, too, but that opens
up the question of what to do if two different publications conflict.
And while I go down that rabbit hole, I wanted to see if anyone thinks
this whole thing is unacceptable. :D
Thanks,
--Jacob
Attachments:
[text/x-patch] WIP-introduce-pg_set_logical_root-for-use-with-pubvi.patch (35.5K, ../../[email protected]/2-WIP-introduce-pg_set_logical_root-for-use-with-pubvi.patch)
download | inline diff:
From 379bf99ea022203d428a4027da753a00a3989c04 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Mon, 26 Sep 2022 13:23:51 -0700
Subject: [PATCH] WIP: introduce pg_set_logical_root for use with pubviaroot
Allows regular inherited tables to be published via their root table,
just like partitions. This works by hijacking pg_inherit's inhseqno
column, and replacing a (single) existing entry for the child with the
value zero, indicating that it should be treated as a logical partition
by the publication machinery.
(For this to work correctly at the moment, the publication must set
publish_via_partition_root=true. See bugs below.)
Initial sync works by pulling in all logical descendants on the
subscriber, then COPYing them one-by-one into the root. The publisher
reuses the existing pubviaroot logic, adding the new logical roots to
code that previously looked only for partition roots.
Known bugs/TODOs:
- When pubviaroot is false, initial sync doesn't work correctly (it
assumes pubviaroot is on and COPYs all descendants into the root).
- The pg_inherits machinery doesn't prohibit changes to inheritance
after an entry has been marked as a logical root.
- I haven't given any thought to interactions with row filters, or to
column lists, or to multiple publications with conflicting pubviaroot
settings.
- pg_set_logical_root() doesn't check for table ownership yet. Anyone
can muck with pg_inherits through it.
- I'm not sure that I'm taking all the necessary locks yet, and those I
do take may be taken in the wrong order.
---
src/backend/catalog/pg_inherits.c | 154 ++++++++++++++
src/backend/catalog/pg_publication.c | 30 +--
src/backend/commands/publicationcmds.c | 10 +
src/backend/replication/logical/tablesync.c | 221 +++++++++++++-------
src/backend/replication/pgoutput/pgoutput.c | 54 ++---
src/include/catalog/pg_inherits.h | 2 +
src/include/catalog/pg_proc.dat | 5 +
src/test/regress/expected/publication.out | 32 +++
src/test/regress/sql/publication.sql | 25 +++
src/test/subscription/t/013_partition.pl | 186 ++++++++++++++++
10 files changed, 608 insertions(+), 111 deletions(-)
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index da969bd2f9..c290a9936a 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -24,10 +24,13 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_inherits.h"
+#include "catalog/partition.h"
#include "parser/parse_type.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/fmgrprotos.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -655,3 +658,154 @@ PartitionHasPendingDetach(Oid partoid)
elog(ERROR, "relation %u is not a partition", partoid);
return false; /* keep compiler quiet */
}
+
+static Oid
+get_logical_parent_worker(Relation inhRel, Oid relid)
+{
+ SysScanDesc scan;
+ ScanKeyData key[2];
+ Oid result = InvalidOid;
+ HeapTuple tuple;
+
+ ScanKeyInit(&key[0],
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 2, key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ Form_pg_inherits form = (Form_pg_inherits) GETSTRUCT(tuple);
+ result = form->inhparent;
+ }
+
+ systable_endscan(scan);
+
+ return result;
+}
+
+static void
+get_logical_ancestors_worker(Relation inhRel, Oid relid, List **ancestors)
+{
+ Oid parentOid;
+
+ /*
+ * Recursion ends at the topmost level, ie., when there's no parent.
+ */
+ parentOid = get_logical_parent_worker(inhRel, relid);
+ if (parentOid == InvalidOid)
+ return;
+
+ *ancestors = lappend_oid(*ancestors, parentOid);
+ get_logical_ancestors_worker(inhRel, parentOid, ancestors);
+}
+
+List *
+get_logical_ancestors(Oid relid, bool is_partition)
+{
+ List *result = NIL;
+ Relation inhRel;
+
+ /* For partitions, this is identical to get_partition_ancestors(). */
+ if (is_partition)
+ return get_partition_ancestors(relid);
+
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
+ get_logical_ancestors_worker(inhRel, relid, &result);
+ table_close(inhRel, AccessShareLock);
+
+ return result;
+}
+
+bool
+has_logical_parent(Relation inhRel, Oid relid)
+{
+ return (get_logical_parent_worker(inhRel, relid) != InvalidOid);
+}
+
+Datum
+pg_set_logical_root(PG_FUNCTION_ARGS)
+{
+ Oid tableoid = PG_GETARG_OID(0);
+ Oid rootoid = PG_GETARG_OID(1);
+ char *tablename;
+ char *rootname;
+ Relation inhRel;
+ ScanKeyData key;
+ SysScanDesc scan;
+ Oid parent = InvalidOid;
+ HeapTuple tuple, copyTuple;
+ Form_pg_inherits form;
+
+ /*
+ * Check that the tables exist.
+ * TODO: check inheritance
+ * TODO: and identical schemas too? or does replication handle that?
+ * TODO: check ownership
+ */
+ tablename = get_rel_name(tableoid);
+ if (tablename == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", tableoid)));
+ rootname = get_rel_name(rootoid);
+ if (rootname == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", rootoid)));
+
+ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
+ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
+
+ /*
+ * We have to make sure that the inheritance relationship already exists,
+ * and that there is only one existing parent for this table.
+ *
+ * TODO: do we have to lock the tables themselves to avoid races?
+ */
+ ScanKeyInit(&key,
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(tableoid));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 1, &key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ form = (Form_pg_inherits) GETSTRUCT(tuple);
+ parent = form->inhparent;
+ copyTuple = heap_copytuple(tuple);
+
+ if (HeapTupleIsValid(systable_getnext(scan)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" inherits from multiple tables",
+ tablename)));
+ }
+
+ if (parent != rootoid)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" does not inherit from intended root table \"%s\"",
+ tablename, rootname)));
+
+ systable_endscan(scan);
+
+ /* Mark the inheritance as a logical root by setting it to zero. */
+ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
+ form->inhseqno = 0;
+
+ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
+
+ heap_freetuple(copyTuple);
+ table_close(inhRel, RowExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index a98fcad421..e1a02e18d9 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -172,11 +172,11 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
}
/*
- * Filter out the partitions whose parent tables were also specified in
+ * Filter out the tables whose logical parent tables were also specified in
* the publication.
*/
static List *
-filter_partitions(List *relids)
+filter_logical_descendants(List *relids)
{
List *result = NIL;
ListCell *lc;
@@ -188,8 +188,7 @@ filter_partitions(List *relids)
List *ancestors = NIL;
Oid relid = lfirst_oid(lc);
- if (get_rel_relispartition(relid))
- ancestors = get_partition_ancestors(relid);
+ ancestors = get_logical_ancestors(relid, get_rel_relispartition(relid));
foreach(lc2, ancestors)
{
@@ -782,11 +781,15 @@ List *
GetAllTablesPublicationRelations(bool pubviaroot)
{
Relation classRel;
+ Relation inhRel;
ScanKeyData key[1];
TableScanDesc scan;
HeapTuple tuple;
List *result = NIL;
+ /* TODO: is there a required order to acquire these locks? */
+ if (pubviaroot)
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
classRel = table_open(RelationRelationId, AccessShareLock);
ScanKeyInit(&key[0],
@@ -802,7 +805,8 @@ GetAllTablesPublicationRelations(bool pubviaroot)
Oid relid = relForm->oid;
if (is_publishable_class(relid, relForm) &&
- !(relForm->relispartition && pubviaroot))
+ !(relForm->relispartition && pubviaroot) &&
+ !(pubviaroot && has_logical_parent(inhRel, relid)))
result = lappend_oid(result, relid);
}
@@ -831,6 +835,9 @@ GetAllTablesPublicationRelations(bool pubviaroot)
}
table_close(classRel, AccessShareLock);
+ if (pubviaroot)
+ table_close(inhRel, AccessShareLock);
+
return result;
}
@@ -1076,15 +1083,14 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
tables = list_concat_unique_oid(relids, schemarelids);
/*
- * If the publication publishes partition changes via their
- * respective root partitioned tables, we must exclude partitions
- * in favor of including the root partitioned tables. Otherwise,
- * the function could return both the child and parent tables
- * which could cause data of the child table to be
- * double-published on the subscriber side.
+ * If the publication publishes table changes via their respective
+ * logical root tables, we must exclude logical descendants in favor
+ * of including the root tables. Otherwise, the function could
+ * return both the child and parent tables which could cause data of
+ * the child table to be double-published on the subscriber side.
*/
if (publication->pubviaroot)
- tables = filter_partitions(tables);
+ tables = filter_logical_descendants(tables);
}
/* Construct a tuple descriptor for the result rows. */
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f4ba572697..7e23bed6c0 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -238,6 +238,8 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* parent table, but the bitmap contains the replica identity
* information of the child table. So, get the column number of the
* child table as parent and child column order could be different.
+ *
+ * TODO: is this applicable to pg_set_logical_root()?
*/
if (context->pubviaroot)
{
@@ -286,6 +288,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
*
* Note that even though the row filter used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
+ *
+ * TODO: is this applicable to pg_set_logical_root()?
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
@@ -336,6 +340,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
* the column list.
*
* Returns true if any replica identity column is not covered by column list.
+ *
+ * TODO: pg_set_logical_root()?
*/
bool
pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
@@ -628,6 +634,8 @@ TransformPubWhereClauses(List *tables, const char *queryString,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
+ * TODO: decide how this interacts with pg_set_logical_root
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ -715,6 +723,8 @@ CheckPubRelationColumnList(char *pubname, List *tables,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
+ * TODO: decide if this interacts with pg_set_logical_root()
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 38dfce7129..b83b9e91ef 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -748,11 +748,12 @@ copy_read_data(void *outbuf, int minread, int maxread)
/*
* Get information about remote relation in similar fashion the RELATION
* message provides during replication. This function also returns the relation
- * qualifications to be used in the COPY command.
+ * qualifications to be used in the COPY command, and the list of tables to COPY
+ * (which for most tables will contain only one entry).
*/
static void
fetch_remote_table_info(char *nspname, char *relname,
- LogicalRepRelation *lrel, List **qual)
+ LogicalRepRelation *lrel, List **qual, List **to_copy)
{
WalRcvExecResult *res;
StringInfoData cmd;
@@ -1063,6 +1064,73 @@ fetch_remote_table_info(char *nspname, char *relname,
walrcv_clear_result(res);
}
+ /*
+ * See if there are any other tables to be copied besides the original. This
+ * happens when a descendant in the inheritance relationship is marked with
+ * pg_set_logical_root().
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000)
+ {
+ Oid descRow[] = {TEXTOID, TEXTOID};
+
+ /*
+ * Find all logical descendants rooted at this table (including the
+ * table itself).
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "WITH RECURSIVE descendants(relid) AS ("
+ " VALUES (%u::oid) UNION ALL"
+ " SELECT inhrelid"
+ " FROM pg_catalog.pg_inherits i, descendants d"
+ " WHERE i.inhseqno = 0"
+ " AND d.relid = i.inhparent"
+ ")"
+ "SELECT n.nspname, c.relname"
+ " FROM descendants,"
+ " pg_catalog.pg_class c"
+ " JOIN pg_catalog.pg_namespace n"
+ " ON (c.relnamespace = n.oid)"
+ " WHERE c.oid = descendants.relid",
+ lrel->remoteid);
+
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(descRow), descRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch logical descendants for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *desc_nspname;
+ char *desc_relname;
+ char *quoted;
+
+ desc_nspname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+ desc_relname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ quoted = quote_qualified_identifier(desc_nspname, desc_relname);
+ *to_copy = lappend(*to_copy, quoted);
+
+ ExecClearTuple(slot);
+ }
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+ }
+ else
+ {
+ /* For older servers, we only COPY the table itself. */
+ char *quoted = quote_qualified_identifier(lrel->nspname,
+ lrel->relname);
+ *to_copy = lappend(*to_copy, quoted);
+ }
+
pfree(cmd.data);
}
@@ -1077,6 +1145,8 @@ copy_table(Relation rel)
LogicalRepRelMapEntry *relmapentry;
LogicalRepRelation lrel;
List *qual = NIL;
+ List *to_copy = NIL;
+ ListCell *cur;
WalRcvExecResult *res;
StringInfoData cmd;
CopyFromState cstate;
@@ -1085,7 +1155,8 @@ copy_table(Relation rel)
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &lrel, &qual,
+ &to_copy);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1094,92 +1165,96 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
- /* Start copy on the publisher. */
- initStringInfo(&cmd);
-
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ foreach(cur, to_copy)
{
- appendStringInfo(&cmd, "COPY %s (",
- quote_qualified_identifier(lrel.nspname, lrel.relname));
+ char *quoted_name = lfirst(cur);
- /*
- * XXX Do we need to list the columns in all cases? Maybe we're
- * replicating all columns?
- */
- for (int i = 0; i < lrel.natts; i++)
- {
- if (i > 0)
- appendStringInfoString(&cmd, ", ");
-
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- }
+ /* Start copy on the publisher. */
+ initStringInfo(&cmd);
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
- else
- {
- /*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
- */
- appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ /* Regular table with no row filter */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
- appendStringInfoString(&cmd, ", ");
- }
+ appendStringInfo(&cmd, "COPY %s (", quoted_name);
- appendStringInfoString(&cmd, " FROM ");
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're
+ * replicating all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
- /*
- * For regular tables, make sure we don't copy data from a child that
- * inherits the named table as those will be copied separately.
- */
- if (lrel.relkind == RELKIND_RELATION)
- appendStringInfoString(&cmd, "ONLY ");
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
- appendStringInfoString(&cmd, quote_qualified_identifier(lrel.nspname, lrel.relname));
- /* list of OR'ed filters */
- if (qual != NIL)
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
+ else
{
- ListCell *lc;
- char *q = strVal(linitial(qual));
+ /*
+ * For non-tables and tables with row filters, we need to do COPY
+ * (SELECT ...), but we can't just do SELECT * because we need to not
+ * copy generated columns. For tables with any row filters, build a
+ * SELECT query with OR'ed row filters for COPY.
+ */
+ appendStringInfoString(&cmd, "COPY (SELECT ");
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
- appendStringInfo(&cmd, " WHERE %s", q);
- for_each_from(lc, qual, 1)
+ appendStringInfoString(&cmd, " FROM ");
+
+ /*
+ * For regular tables, make sure we don't copy data from a child that
+ * inherits the named table as those will be copied separately.
+ */
+ if (lrel.relkind == RELKIND_RELATION)
+ appendStringInfoString(&cmd, "ONLY ");
+
+ appendStringInfoString(&cmd, quoted_name);
+ /* list of OR'ed filters */
+ if (qual != NIL)
{
- q = strVal(lfirst(lc));
- appendStringInfo(&cmd, " OR %s", q);
+ ListCell *lc;
+ char *q = strVal(linitial(qual));
+
+ appendStringInfo(&cmd, " WHERE %s", q);
+ for_each_from(lc, qual, 1)
+ {
+ q = strVal(lfirst(lc));
+ appendStringInfo(&cmd, " OR %s", q);
+ }
+ list_free_deep(qual);
}
- list_free_deep(qual);
- }
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
- res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
- pfree(cmd.data);
- if (res->status != WALRCV_OK_COPY_OUT)
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("could not start initial contents copy for table \"%s.%s\": %s",
- lrel.nspname, lrel.relname, res->err)));
- walrcv_clear_result(res);
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
+ pfree(cmd.data);
+ if (res->status != WALRCV_OK_COPY_OUT)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not start initial contents copy for table \"%s.%s\" from remote %s: %s",
+ lrel.nspname, lrel.relname, quoted_name, res->err)));
+ walrcv_clear_result(res);
- copybuf = makeStringInfo();
+ copybuf = makeStringInfo();
- pstate = make_parsestate(NULL);
- (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
- NULL, false, false);
+ pstate = make_parsestate(NULL);
+ (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
+ NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
- cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, NIL);
+ attnamelist = make_copy_attnamelist(relmapentry);
+ cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, NIL);
- /* Do the copy */
- (void) CopyFrom(cstate);
+ /* Do the copy */
+ (void) CopyFrom(cstate);
+ }
logicalrep_rel_close(relmapentry, NoLock);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 1a80d67bb9..897c5e0d92 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -14,6 +14,7 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
+#include "catalog/pg_inherits.h"
#include "catalog/pg_publication.h"
#include "catalog/pg_publication_rel.h"
#include "catalog/pg_subscription.h"
@@ -1450,7 +1451,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Switch relation if publishing via root. */
if (relentry->publish_as_relid != RelationGetRelid(relation))
{
- Assert(relation->rd_rel->relispartition);
ancestor = RelationIdGetRelation(relentry->publish_as_relid);
targetrel = ancestor;
/* Convert tuple if needed. */
@@ -2141,55 +2141,57 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
int ancestor_level = 0;
/*
- * If this is a FOR ALL TABLES publication, pick the partition
+ * If this is a FOR ALL TABLES publication, pick the logical
* root and set the ancestor level accordingly.
*/
if (pub->alltables)
{
publish = true;
- if (pub->pubviaroot && am_partition)
+ if (pub->pubviaroot)
{
- List *ancestors = get_partition_ancestors(relid);
+ List *ancestors;
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ if (ancestors != NIL)
+ {
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
}
if (!publish)
{
bool ancestor_published = false;
+ Oid ancestor;
+ int level;
+ List *ancestors;
/*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
+ * Check if any of the logical ancestors (that is, partition
+ * parents or tables marked with pg_set_logical_root()) are
+ * published. If so, note down the topmost ancestor that is
* published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
+ * relation via which to publish this table's changes.
*/
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
-
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
- if (ancestor != InvalidOid)
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
{
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
+ pub_relid = ancestor;
+ ancestor_level = level;
}
}
if (list_member_oid(pubids, pub->oid) ||
list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
+ (am_partition && ancestor_published))
publish = true;
}
diff --git a/src/include/catalog/pg_inherits.h b/src/include/catalog/pg_inherits.h
index ce154ab943..59d72a97b8 100644
--- a/src/include/catalog/pg_inherits.h
+++ b/src/include/catalog/pg_inherits.h
@@ -63,5 +63,7 @@ extern bool DeleteInheritsTuple(Oid inhrelid, Oid inhparent,
bool expect_detach_pending,
const char *childname);
extern bool PartitionHasPendingDetach(Oid partoid);
+extern List *get_logical_ancestors(Oid relid, bool is_partition);
+extern bool has_logical_parent(Relation inhRel, Oid relid);
#endif /* PG_INHERITS_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 86eb8e8c58..6f5f85c5cc 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11891,4 +11891,9 @@
prorettype => 'bytea', proargtypes => 'pg_brin_minmax_multi_summary',
prosrc => 'brin_minmax_multi_summary_send' },
+{ oid => '8136', descr => 'mark a table root for logical replication',
+ proname => 'pg_set_logical_root', provolatile => 'v', proparallel => 'u',
+ prorettype => 'void', proargtypes => 'regclass regclass',
+ prosrc => 'pg_set_logical_root' },
+
]
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 427f87ea07..0e06103f4c 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -1718,9 +1718,41 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+-- Sanity check cases for pg_set_logical_root().
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+-- marking roots between unrelated tables is not allowed
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+ERROR: table "ipart1" does not inherit from intended root table "iroot"
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
+ERROR: table "ipart2" does not inherit from intended root table "tbl1"
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+ERROR: table "ipart2" inherits from multiple tables
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index a47c5939d5..52a0d6ba48 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -1087,9 +1087,34 @@ ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
+-- Sanity check cases for pg_set_logical_root().
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+
+-- marking roots between unrelated tables is not allowed
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
+
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+
+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
+
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 11a5c3c03e..b6e3143536 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -877,4 +877,190 @@ $result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab5_1 ORDER BY 1");
is($result, qq(4||1), 'updates of tab5 replicated correctly');
+# Test that replication works for older inheritance/trigger setups as well.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab1_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab1_1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO itab1_2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab1_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab1_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (CHECK (a = 1)) INHERITS (itab2)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab2_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab2_1 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab2_trigger
+ BEFORE INSERT ON itab2
+ FOR EACH ROW EXECUTE FUNCTION itab2_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab2_1', 'itab2')");
+
+# itab2_1 should be published using its own identity here, since its parent is
+# not included. itab1_1 should be published via its parent, itab1, without
+# duplicating the rows.
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub_viaroot ADD TABLE itab1, itab1_1, itab2_1");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (2, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (0, 'itab2')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2')");
+
+# Subscriber 1 only subscribes to some of the partitions, and does not set up
+# partition triggers, to check for the correct routing.
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (a int, b text)");
+
+# Subscriber 2 has different partition names for itab1, and it doesn't partition
+# itab2 at all.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_subscriber2->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO public.itab1_part1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO public.itab1_part2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_subscriber2->safe_psql('postgres', "
+ CREATE TRIGGER itab_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab_trigger();");
+$node_subscriber2->safe_psql('postgres', "
+ ALTER TABLE itab1 ENABLE ALWAYS TRIGGER itab_trigger;");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot REFRESH PUBLICATION");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 REFRESH PUBLICATION");
+
+$node_subscriber1->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
+
+# check that data is synced correctly
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 1');
+
+# all of the data should have been routed to itab1 directly (there are no
+# triggers on subscriber 1 to move it elsewhere)
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data correctly routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2), 'initial data synced for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1), 'initial data correctly routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2), 'initial data synced for itab2 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1-new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2-new')");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+$node_publisher->wait_for_catchup('sub2');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2
+1|itab2-new), 'new data routed for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_part1 ORDER BY 1, 2");
+is($result, qq(1|itab1
+1|itab1-new), 'new data moved to itab1_part1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2
+1|itab2-new), 'new data routed for itab2 on subscriber 2');
+
done_testing();
--
2.25.1
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-02-28 22:47 ` Jacob Champion <[email protected]>
2023-03-07 10:40 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-03-31 22:17 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
2023-04-04 03:53 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
0 siblings, 3 replies; 18+ messages in thread
From: Jacob Champion @ 2023-02-28 22:47 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>
Hi,
I'm going to register this in CF for feedback.
Summary for potential reviewers: we don't use declarative partitions in
the Timescale partitioning scheme, but it'd be really nice to be able to
replicate between our tables and standard tables, or between two
Timescale-partitioned tables with different layouts. This patch lets
extensions (or savvy users) upgrade an existing inheritance relationship
between two tables into a "logical partition" relationship, so that they
can be handled with the publish_via_partition_root machinery.
I hope this might also help pg_partman users migrate between old- and
new-style partition schemes, but that's speculation.
On 1/20/23 09:53, Jacob Champion wrote:
>> 2) While this strategy works well for ongoing replication, it's not
>> enough to get the initial synchronization correct. The subscriber
>> still does a COPY of the root table directly, missing out on all the
>> logical descendant data. The publisher will have to tell the
>> subscriber about the relationship somehow, and older subscriber
>> versions won't understand how to use that (similar to how old
>> subscribers can't correctly handle row filters).
>
> I partially solved this by having the subscriber pull the logical
> hierarchy from the publisher to figure out which tables to COPY. This
> works when publish_via_partition_root=true, but it doesn't correctly
> return to the previous behavior when the setting is false. I need to
> check the publication setting from the subscriber, too, but that opens
> up the question of what to do if two different publications conflict.
Second draft attached, which fixes that bug. I kept thinking to myself
that this would be much easier if the publisher told the subscriber what
data to copy rather than having the subscriber hardcode the initial sync
process... and then I realized that I could, sort of, move in that
direction.
This version adds a SQL function to determine the list of source tables
to COPY into a subscriber's target table. Now the publisher can make use
of whatever catalogs it needs to make that list and the subscriber
doesn't need to couple to them. (This could also provide a way for
publishers to provide more generic "table indirection" in the future,
but I'm wary of selling genericism as a feature here.)
I haven't solved the problem where two publications of the same table
have different settings for publish_via_partition_root. I was curious to
see how the existing partition code prevented problems, but I'm not
really sure that it does... Here are some situations where the existing
implementation duplicates data on the initial sync:
1) A single subscription to two publications, one with
publish_via_partition_root on and the other off, which publish the same
partitioned table
2) A single subscription to two publications with
publish_via_partition_root on, one of which publishes a root partition
and the other of which publishes a descendant/leaf
3) A single subscription to two publications with
publish_via_partition_root on, one of which publishes FOR ALL TABLES and
the other of which publishes a descendant/leaf
Is it expected that DBAs should avoid these cases, or are they worth
pursuing with a bug fix?
Thanks,
--Jacob
Attachments:
[text/x-patch] WIP-introduce-pg_set_logical_root-for-use-with-pubvi.patch (46.7K, ../../[email protected]/2-WIP-introduce-pg_set_logical_root-for-use-with-pubvi.patch)
download | inline diff:
From 6d31d13f9d7c400cf717426c4f336f7ad19a4391 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Mon, 26 Sep 2022 13:23:51 -0700
Subject: [PATCH] WIP: introduce pg_set_logical_root for use with pubviaroot
Allows regular inherited tables to be published via their root table,
just like partitions. This works by hijacking pg_inherit's inhseqno
column, and replacing a (single) existing entry for the child with the
value zero, indicating that it should be treated as a logical partition
by the publication machinery.
Initial sync works by asking the publisher for a list of logical
descendants of the published table, then COPYing them one-by-one into
the root. The publisher reuses the existing pubviaroot logic, adding the
new logical roots to code that previously looked only for partition
roots.
Known bugs/TODOs:
- The pg_inherits machinery doesn't prohibit changes to inheritance
after an entry has been marked as a logical root.
- I haven't given any thought to interactions with row filters, or to
column lists, or to multiple publications with conflicting pubviaroot
settings.
- pg_set_logical_root() doesn't check for table ownership yet. Anyone
can muck with pg_inherits through it.
- I'm not sure that I'm taking all the necessary locks yet, and those I
do take may be taken in the wrong order.
---
src/backend/catalog/pg_inherits.c | 202 ++++++++++++++++-
src/backend/catalog/pg_publication.c | 120 +++++++++-
src/backend/commands/publicationcmds.c | 10 +
src/backend/partitioning/partdesc.c | 3 +-
src/backend/replication/logical/tablesync.c | 235 +++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 54 ++---
src/include/catalog/pg_inherits.h | 7 +-
src/include/catalog/pg_proc.dat | 10 +
src/test/regress/expected/publication.out | 32 +++
src/test/regress/sql/publication.sql | 25 ++
src/test/subscription/t/013_partition.pl | 239 ++++++++++++++++++++
11 files changed, 817 insertions(+), 120 deletions(-)
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index da969bd2f9..659f5125f1 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -24,14 +24,20 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_inherits.h"
+#include "catalog/partition.h"
#include "parser/parse_type.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/fmgrprotos.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+static List * find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only);
+
/*
* Entry of a hash table used in find_all_inheritors. See below.
*/
@@ -59,14 +65,14 @@ List *
find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
{
return find_inheritance_children_extended(parentrelId, true, lockmode,
- NULL, NULL);
+ NULL, NULL, false);
}
/*
* find_inheritance_children_extended
*
* As find_inheritance_children, with more options regarding detached
- * partitions.
+ * partitions and logical roots.
*
* If a partition's pg_inherits row is marked "detach pending",
* *detached_exist (if not null) is set true.
@@ -78,16 +84,21 @@ find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
* whether the transaction that marked those partitions as detached appears
* committed to the active snapshot. In addition, *detached_xmin (if not null)
* is set to the xmin of the row of the detached partition.
+ *
+ * If logical_only is true, only tables marked explicitly via
+ * pg_set_logical_root() are included in the output list.
*/
List *
find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
LOCKMODE lockmode, bool *detached_exist,
- TransactionId *detached_xmin)
+ TransactionId *detached_xmin,
+ bool logical_only)
{
List *list = NIL;
Relation relation;
+ Oid index;
SysScanDesc scan;
- ScanKeyData key[1];
+ ScanKeyData key[2];
HeapTuple inheritsTuple;
Oid inhrelid;
Oid *oidarr;
@@ -110,14 +121,24 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
numoids = 0;
relation = table_open(InheritsRelationId, AccessShareLock);
+ index = InheritsParentIndexId;
ScanKeyInit(&key[0],
Anum_pg_inherits_inhparent,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(parentrelId));
- scan = systable_beginscan(relation, InheritsParentIndexId, true,
- NULL, 1, key);
+ if (logical_only)
+ {
+ index = InheritsParentSeqnoIndexId;
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+ }
+
+ scan = systable_beginscan(relation, index, true,
+ NULL, logical_only ? 2 : 1, key);
while ((inheritsTuple = systable_getnext(scan)) != NULL)
{
@@ -254,6 +275,20 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
*/
List *
find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
+{
+ return find_all_inheritors_internal(parentrelId, lockmode, numparents,
+ false);
+}
+
+List *
+find_all_logical_inheritors(Oid parentrelId)
+{
+ return find_all_inheritors_internal(parentrelId, NoLock, NULL, true);
+}
+
+static List *
+find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only)
{
/* hash table for O(1) rel_oid -> rel_numparents cell lookup */
HTAB *seen_rels;
@@ -290,7 +325,9 @@ find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
ListCell *lc;
/* Get the direct children of this rel */
- currentchildren = find_inheritance_children(currentrel, lockmode);
+ currentchildren =
+ find_inheritance_children_extended(currentrel, true, lockmode,
+ NULL, NULL, logical_only);
/*
* Add to the queue only those children not already seen. This avoids
@@ -655,3 +692,154 @@ PartitionHasPendingDetach(Oid partoid)
elog(ERROR, "relation %u is not a partition", partoid);
return false; /* keep compiler quiet */
}
+
+static Oid
+get_logical_parent_worker(Relation inhRel, Oid relid)
+{
+ SysScanDesc scan;
+ ScanKeyData key[2];
+ Oid result = InvalidOid;
+ HeapTuple tuple;
+
+ ScanKeyInit(&key[0],
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 2, key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ Form_pg_inherits form = (Form_pg_inherits) GETSTRUCT(tuple);
+ result = form->inhparent;
+ }
+
+ systable_endscan(scan);
+
+ return result;
+}
+
+static void
+get_logical_ancestors_worker(Relation inhRel, Oid relid, List **ancestors)
+{
+ Oid parentOid;
+
+ /*
+ * Recursion ends at the topmost level, ie., when there's no parent.
+ */
+ parentOid = get_logical_parent_worker(inhRel, relid);
+ if (parentOid == InvalidOid)
+ return;
+
+ *ancestors = lappend_oid(*ancestors, parentOid);
+ get_logical_ancestors_worker(inhRel, parentOid, ancestors);
+}
+
+List *
+get_logical_ancestors(Oid relid, bool is_partition)
+{
+ List *result = NIL;
+ Relation inhRel;
+
+ /* For partitions, this is identical to get_partition_ancestors(). */
+ if (is_partition)
+ return get_partition_ancestors(relid);
+
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
+ get_logical_ancestors_worker(inhRel, relid, &result);
+ table_close(inhRel, AccessShareLock);
+
+ return result;
+}
+
+bool
+has_logical_parent(Relation inhRel, Oid relid)
+{
+ return (get_logical_parent_worker(inhRel, relid) != InvalidOid);
+}
+
+Datum
+pg_set_logical_root(PG_FUNCTION_ARGS)
+{
+ Oid tableoid = PG_GETARG_OID(0);
+ Oid rootoid = PG_GETARG_OID(1);
+ char *tablename;
+ char *rootname;
+ Relation inhRel;
+ ScanKeyData key;
+ SysScanDesc scan;
+ Oid parent = InvalidOid;
+ HeapTuple tuple, copyTuple;
+ Form_pg_inherits form;
+
+ /*
+ * Check that the tables exist.
+ * TODO: check inheritance
+ * TODO: and identical schemas too? or does replication handle that?
+ * TODO: check ownership
+ */
+ tablename = get_rel_name(tableoid);
+ if (tablename == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", tableoid)));
+ rootname = get_rel_name(rootoid);
+ if (rootname == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", rootoid)));
+
+ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
+ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
+
+ /*
+ * We have to make sure that the inheritance relationship already exists,
+ * and that there is only one existing parent for this table.
+ *
+ * TODO: do we have to lock the tables themselves to avoid races?
+ */
+ ScanKeyInit(&key,
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(tableoid));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 1, &key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ form = (Form_pg_inherits) GETSTRUCT(tuple);
+ parent = form->inhparent;
+ copyTuple = heap_copytuple(tuple);
+
+ if (HeapTupleIsValid(systable_getnext(scan)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" inherits from multiple tables",
+ tablename)));
+ }
+
+ if (parent != rootoid)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" does not inherit from intended root table \"%s\"",
+ tablename, rootname)));
+
+ systable_endscan(scan);
+
+ /* Mark the inheritance as a logical root by setting it to zero. */
+ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
+ form->inhseqno = 0;
+
+ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
+
+ heap_freetuple(copyTuple);
+ table_close(inhRel, RowExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index a98fcad421..04f7dc8099 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -172,11 +172,11 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
}
/*
- * Filter out the partitions whose parent tables were also specified in
+ * Filter out the tables whose logical parent tables were also specified in
* the publication.
*/
static List *
-filter_partitions(List *relids)
+filter_logical_descendants(List *relids)
{
List *result = NIL;
ListCell *lc;
@@ -188,8 +188,7 @@ filter_partitions(List *relids)
List *ancestors = NIL;
Oid relid = lfirst_oid(lc);
- if (get_rel_relispartition(relid))
- ancestors = get_partition_ancestors(relid);
+ ancestors = get_logical_ancestors(relid, get_rel_relispartition(relid));
foreach(lc2, ancestors)
{
@@ -782,11 +781,15 @@ List *
GetAllTablesPublicationRelations(bool pubviaroot)
{
Relation classRel;
+ Relation inhRel;
ScanKeyData key[1];
TableScanDesc scan;
HeapTuple tuple;
List *result = NIL;
+ /* TODO: is there a required order to acquire these locks? */
+ if (pubviaroot)
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
classRel = table_open(RelationRelationId, AccessShareLock);
ScanKeyInit(&key[0],
@@ -802,7 +805,8 @@ GetAllTablesPublicationRelations(bool pubviaroot)
Oid relid = relForm->oid;
if (is_publishable_class(relid, relForm) &&
- !(relForm->relispartition && pubviaroot))
+ !(relForm->relispartition && pubviaroot) &&
+ !(pubviaroot && has_logical_parent(inhRel, relid)))
result = lappend_oid(result, relid);
}
@@ -831,6 +835,9 @@ GetAllTablesPublicationRelations(bool pubviaroot)
}
table_close(classRel, AccessShareLock);
+ if (pubviaroot)
+ table_close(inhRel, AccessShareLock);
+
return result;
}
@@ -1076,15 +1083,14 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
tables = list_concat_unique_oid(relids, schemarelids);
/*
- * If the publication publishes partition changes via their
- * respective root partitioned tables, we must exclude partitions
- * in favor of including the root partitioned tables. Otherwise,
- * the function could return both the child and parent tables
- * which could cause data of the child table to be
- * double-published on the subscriber side.
+ * If the publication publishes table changes via their respective
+ * logical root tables, we must exclude logical descendants in favor
+ * of including the root tables. Otherwise, the function could
+ * return both the child and parent tables which could cause data of
+ * the child table to be double-published on the subscriber side.
*/
if (publication->pubviaroot)
- tables = filter_partitions(tables);
+ tables = filter_logical_descendants(tables);
}
/* Construct a tuple descriptor for the result rows. */
@@ -1190,3 +1196,93 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Returns a list of tables that should be copied during initial sync for the
+ * given root table and publication.
+ *
+ * The only time this list consists of anything more than the table itself is
+ * when a publication's publish_via_partition_root is set to true and the table
+ * has inherited child tables in the publication that have been marked with
+ * pg_set_logical_root().
+ */
+Datum
+pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
+{
+#define NUM_SYNC_TABLES_ELEM 1
+ FuncCallContext *funcctx;
+ Oid rootid = PG_GETARG_OID(0);
+ char *pubname = text_to_cstring(PG_GETARG_TEXT_PP(1));
+ List *tables;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+ Publication *publication;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ publication = GetPublicationByName(pubname, false);
+
+ /* TODO: do the tables in this list need to be locked? */
+ if (publication->pubviaroot)
+ tables = find_all_logical_inheritors(rootid);
+ else
+ tables = list_make1_oid(rootid);
+
+ if (!publication->alltables)
+ {
+ List *relids;
+ List *schemarelids;
+ List *published;
+ List *result = NIL;
+ ListCell *cell;
+
+ /*
+ * Filter our list of tables to those that are actually included in
+ * the publication.
+ */
+ relids = GetPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_ALL);
+ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_LEAF);
+ published = list_concat_unique_oid(relids, schemarelids);
+
+ foreach(cell, tables)
+ {
+ Oid current = lfirst_oid(cell);
+
+ if (list_member_oid(published, current))
+ result = lappend_oid(result, current);
+ }
+
+ tables = result;
+ }
+
+ funcctx->user_fctx = (void *) tables;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+ tables = (List *) funcctx->user_fctx;
+
+ if (funcctx->call_cntr < list_length(tables))
+ {
+ Oid relid = list_nth_oid(tables, funcctx->call_cntr);
+
+ SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ }
+
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f4ba572697..7e23bed6c0 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -238,6 +238,8 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* parent table, but the bitmap contains the replica identity
* information of the child table. So, get the column number of the
* child table as parent and child column order could be different.
+ *
+ * TODO: is this applicable to pg_set_logical_root()?
*/
if (context->pubviaroot)
{
@@ -286,6 +288,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
*
* Note that even though the row filter used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
+ *
+ * TODO: is this applicable to pg_set_logical_root()?
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
@@ -336,6 +340,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
* the column list.
*
* Returns true if any replica identity column is not covered by column list.
+ *
+ * TODO: pg_set_logical_root()?
*/
bool
pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
@@ -628,6 +634,8 @@ TransformPubWhereClauses(List *tables, const char *queryString,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
+ * TODO: decide how this interacts with pg_set_logical_root
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ -715,6 +723,8 @@ CheckPubRelationColumnList(char *pubname, List *tables,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
+ * TODO: decide if this interacts with pg_set_logical_root()
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
diff --git a/src/backend/partitioning/partdesc.c b/src/backend/partitioning/partdesc.c
index de06caccd2..2e34983117 100644
--- a/src/backend/partitioning/partdesc.c
+++ b/src/backend/partitioning/partdesc.c
@@ -162,7 +162,8 @@ RelationBuildPartitionDesc(Relation rel, bool omit_detached)
inhoids = find_inheritance_children_extended(RelationGetRelid(rel),
omit_detached, NoLock,
&detached_exist,
- &detached_xmin);
+ &detached_xmin,
+ false);
nparts = list_length(inhoids);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 07eea504ba..14dcf2a825 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -756,11 +756,12 @@ copy_read_data(void *outbuf, int minread, int maxread)
/*
* Get information about remote relation in similar fashion the RELATION
* message provides during replication. This function also returns the relation
- * qualifications to be used in the COPY command.
+ * qualifications to be used in the COPY command, and the list of tables to COPY
+ * (which for most tables will contain only one entry).
*/
static void
fetch_remote_table_info(char *nspname, char *relname,
- LogicalRepRelation *lrel, List **qual)
+ LogicalRepRelation *lrel, List **qual, List **to_copy)
{
WalRcvExecResult *res;
StringInfoData cmd;
@@ -1071,6 +1072,87 @@ fetch_remote_table_info(char *nspname, char *relname,
walrcv_clear_result(res);
}
+ /*
+ * See if there are any other tables to be copied besides the original. This
+ * happens when a descendant in the inheritance relationship is marked with
+ * pg_set_logical_root() and is part of a publication with
+ * publish_via_partition_root = true.
+ *
+ * FIXME: if two publications have conflicting settings for
+ * publish_via_partition_root, this behaves strangely. It looks like that's
+ * true for regular partitions too...
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000)
+ {
+ Oid descRow[] = {TEXTOID, TEXTOID};
+ StringInfoData pub_names;
+
+ /* Build the pubname list. */
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ char *pubname = strVal(lfirst(lc));
+
+ if (foreach_current_index(lc) > 0)
+ appendStringInfoString(&pub_names, ", ");
+
+ appendStringInfoString(&pub_names, quote_literal_cstr(pubname));
+ }
+
+ /*
+ * Ask the server what it wants us to sync.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT n.nspname, c.relname"
+ " FROM pg_catalog.pg_publication p,"
+ " pg_catalog.pg_class c"
+ " JOIN pg_catalog.pg_namespace n"
+ " ON (c.relnamespace = n.oid)"
+ " JOIN LATERAL pg_catalog.pg_get_publication_rels_to_sync(%u::regclass, p.pubname) relid"
+ " ON (c.oid = relid)"
+ " WHERE p.pubname IN ( %s )",
+ lrel->remoteid, pub_names.data);
+
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(descRow), descRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch logical descendants for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *desc_nspname;
+ char *desc_relname;
+ char *quoted;
+
+ desc_nspname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+ desc_relname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ quoted = quote_qualified_identifier(desc_nspname, desc_relname);
+ *to_copy = lappend(*to_copy, quoted);
+
+ ExecClearTuple(slot);
+ }
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+
+ pfree(pub_names.data);
+ }
+ else
+ {
+ /* For older servers, we only COPY the table itself. */
+ char *quoted = quote_qualified_identifier(lrel->nspname,
+ lrel->relname);
+ *to_copy = lappend(*to_copy, quoted);
+ }
+
pfree(cmd.data);
}
@@ -1085,6 +1167,8 @@ copy_table(Relation rel)
LogicalRepRelMapEntry *relmapentry;
LogicalRepRelation lrel;
List *qual = NIL;
+ List *to_copy = NIL;
+ ListCell *cur;
WalRcvExecResult *res;
StringInfoData cmd;
CopyFromState cstate;
@@ -1093,7 +1177,8 @@ copy_table(Relation rel)
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &lrel, &qual,
+ &to_copy);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1102,92 +1187,96 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
- /* Start copy on the publisher. */
- initStringInfo(&cmd);
-
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ foreach(cur, to_copy)
{
- appendStringInfo(&cmd, "COPY %s (",
- quote_qualified_identifier(lrel.nspname, lrel.relname));
+ char *quoted_name = lfirst(cur);
- /*
- * XXX Do we need to list the columns in all cases? Maybe we're
- * replicating all columns?
- */
- for (int i = 0; i < lrel.natts; i++)
- {
- if (i > 0)
- appendStringInfoString(&cmd, ", ");
+ /* Start copy on the publisher. */
+ initStringInfo(&cmd);
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- }
-
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
- else
- {
- /*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
- */
- appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ /* Regular table with no row filter */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
- appendStringInfoString(&cmd, ", ");
- }
+ appendStringInfo(&cmd, "COPY %s (", quoted_name);
- appendStringInfoString(&cmd, " FROM ");
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're
+ * replicating all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
- /*
- * For regular tables, make sure we don't copy data from a child that
- * inherits the named table as those will be copied separately.
- */
- if (lrel.relkind == RELKIND_RELATION)
- appendStringInfoString(&cmd, "ONLY ");
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
- appendStringInfoString(&cmd, quote_qualified_identifier(lrel.nspname, lrel.relname));
- /* list of OR'ed filters */
- if (qual != NIL)
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
+ else
{
- ListCell *lc;
- char *q = strVal(linitial(qual));
+ /*
+ * For non-tables and tables with row filters, we need to do COPY
+ * (SELECT ...), but we can't just do SELECT * because we need to not
+ * copy generated columns. For tables with any row filters, build a
+ * SELECT query with OR'ed row filters for COPY.
+ */
+ appendStringInfoString(&cmd, "COPY (SELECT ");
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+
+ appendStringInfoString(&cmd, " FROM ");
- appendStringInfo(&cmd, " WHERE %s", q);
- for_each_from(lc, qual, 1)
+ /*
+ * For regular tables, make sure we don't copy data from a child that
+ * inherits the named table as those will be copied separately.
+ */
+ if (lrel.relkind == RELKIND_RELATION)
+ appendStringInfoString(&cmd, "ONLY ");
+
+ appendStringInfoString(&cmd, quoted_name);
+ /* list of OR'ed filters */
+ if (qual != NIL)
{
- q = strVal(lfirst(lc));
- appendStringInfo(&cmd, " OR %s", q);
+ ListCell *lc;
+ char *q = strVal(linitial(qual));
+
+ appendStringInfo(&cmd, " WHERE %s", q);
+ for_each_from(lc, qual, 1)
+ {
+ q = strVal(lfirst(lc));
+ appendStringInfo(&cmd, " OR %s", q);
+ }
+ list_free_deep(qual);
}
- list_free_deep(qual);
- }
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
- res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
- pfree(cmd.data);
- if (res->status != WALRCV_OK_COPY_OUT)
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("could not start initial contents copy for table \"%s.%s\": %s",
- lrel.nspname, lrel.relname, res->err)));
- walrcv_clear_result(res);
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
+ pfree(cmd.data);
+ if (res->status != WALRCV_OK_COPY_OUT)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not start initial contents copy for table \"%s.%s\" from remote %s: %s",
+ lrel.nspname, lrel.relname, quoted_name, res->err)));
+ walrcv_clear_result(res);
- copybuf = makeStringInfo();
+ copybuf = makeStringInfo();
- pstate = make_parsestate(NULL);
- (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
- NULL, false, false);
+ pstate = make_parsestate(NULL);
+ (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
+ NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
- cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, NIL);
+ attnamelist = make_copy_attnamelist(relmapentry);
+ cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, NIL);
- /* Do the copy */
- (void) CopyFrom(cstate);
+ /* Do the copy */
+ (void) CopyFrom(cstate);
+ }
logicalrep_rel_close(relmapentry, NoLock);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 0df1acbb7a..7cfb6130a8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -14,6 +14,7 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
+#include "catalog/pg_inherits.h"
#include "catalog/pg_publication.h"
#include "catalog/pg_publication_rel.h"
#include "catalog/pg_subscription.h"
@@ -1454,7 +1455,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Switch relation if publishing via root. */
if (relentry->publish_as_relid != RelationGetRelid(relation))
{
- Assert(relation->rd_rel->relispartition);
ancestor = RelationIdGetRelation(relentry->publish_as_relid);
targetrel = ancestor;
/* Convert tuple if needed. */
@@ -2149,55 +2149,57 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
int ancestor_level = 0;
/*
- * If this is a FOR ALL TABLES publication, pick the partition
+ * If this is a FOR ALL TABLES publication, pick the logical
* root and set the ancestor level accordingly.
*/
if (pub->alltables)
{
publish = true;
- if (pub->pubviaroot && am_partition)
+ if (pub->pubviaroot)
{
- List *ancestors = get_partition_ancestors(relid);
+ List *ancestors;
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ if (ancestors != NIL)
+ {
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
}
if (!publish)
{
bool ancestor_published = false;
+ Oid ancestor;
+ int level;
+ List *ancestors;
/*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
+ * Check if any of the logical ancestors (that is, partition
+ * parents or tables marked with pg_set_logical_root()) are
+ * published. If so, note down the topmost ancestor that is
* published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
+ * relation via which to publish this table's changes.
*/
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
-
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
- if (ancestor != InvalidOid)
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
{
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
+ pub_relid = ancestor;
+ ancestor_level = level;
}
}
if (list_member_oid(pubids, pub->oid) ||
list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
+ (am_partition && ancestor_published))
publish = true;
}
diff --git a/src/include/catalog/pg_inherits.h b/src/include/catalog/pg_inherits.h
index ce154ab943..cfcacacf76 100644
--- a/src/include/catalog/pg_inherits.h
+++ b/src/include/catalog/pg_inherits.h
@@ -46,14 +46,17 @@ typedef FormData_pg_inherits *Form_pg_inherits;
DECLARE_UNIQUE_INDEX_PKEY(pg_inherits_relid_seqno_index, 2680, InheritsRelidSeqnoIndexId, on pg_inherits using btree(inhrelid oid_ops, inhseqno int4_ops));
DECLARE_INDEX(pg_inherits_parent_index, 2187, InheritsParentIndexId, on pg_inherits using btree(inhparent oid_ops));
+DECLARE_INDEX(pg_inherits_parent_seqno_index, 8138, InheritsParentSeqnoIndexId, on pg_inherits using btree(inhparent oid_ops, inhseqno int4_ops));
extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);
extern List *find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
- LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin);
+ LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin,
+ bool logical_only);
extern List *find_all_inheritors(Oid parentrelId, LOCKMODE lockmode,
List **numparents);
+extern List *find_all_logical_inheritors(Oid parentrelId);
extern bool has_subclass(Oid relationId);
extern bool has_superclass(Oid relationId);
extern bool typeInheritsFrom(Oid subclassTypeId, Oid superclassTypeId);
@@ -63,5 +66,7 @@ extern bool DeleteInheritsTuple(Oid inhrelid, Oid inhparent,
bool expect_detach_pending,
const char *childname);
extern bool PartitionHasPendingDetach(Oid partoid);
+extern List *get_logical_ancestors(Oid relid, bool is_partition);
+extern bool has_logical_parent(Relation inhRel, Oid relid);
#endif /* PG_INHERITS_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 505595620e..9d7b3c11bf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11729,6 +11729,11 @@
proname => 'pg_relation_is_publishable', provolatile => 's',
prorettype => 'bool', proargtypes => 'regclass',
prosrc => 'pg_relation_is_publishable' },
+{ oid => '8137', descr => 'get list of tables to copy during initial sync',
+ proname => 'pg_get_publication_rels_to_sync', prorows => '10', proretset => 't',
+ provolatile => 's', prorettype => 'regclass', proargtypes => 'regclass text',
+ proargnames => '{rootid,pubname}',
+ prosrc => 'pg_get_publication_rels_to_sync' },
# rls
{ oid => '3298',
@@ -11887,6 +11892,11 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+{ oid => '8136', descr => 'mark a table root for logical replication',
+ proname => 'pg_set_logical_root', provolatile => 'v', proparallel => 'u',
+ prorettype => 'void', proargtypes => 'regclass regclass',
+ prosrc => 'pg_set_logical_root' },
+
{ oid => '4350', descr => 'Unicode normalization',
proname => 'normalize', prorettype => 'text', proargtypes => 'text text',
prosrc => 'unicode_normalize_func' },
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 427f87ea07..0e06103f4c 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -1718,9 +1718,41 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+-- Sanity check cases for pg_set_logical_root().
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+-- marking roots between unrelated tables is not allowed
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+ERROR: table "ipart1" does not inherit from intended root table "iroot"
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
+ERROR: table "ipart2" does not inherit from intended root table "tbl1"
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+ERROR: table "ipart2" inherits from multiple tables
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index a47c5939d5..52a0d6ba48 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -1087,9 +1087,34 @@ ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
+-- Sanity check cases for pg_set_logical_root().
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+
+-- marking roots between unrelated tables is not allowed
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
+
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+
+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
+
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 11a5c3c03e..a4d771f31f 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -388,6 +388,59 @@ $node_subscriber1->append_conf('postgresql.conf',
"log_min_messages = warning");
$node_subscriber1->reload;
+# Make sure standard inheritance setups aren't broken by the new
+# pg_set_logical_root() handling.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (LIKE itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (LIKE itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_2 VALUES (2, 'itab1')");
+
+# Regression: Create a publication for an unrelated table and set
+# publish_via_partition_root. This should have no effect at all, since itab1
+# isn't supposed to be using this publication...
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION dummy_pub FOR TABLE tab1 WITH (publish_via_partition_root = true)");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 ADD PUBLICATION dummy_pub");
+$node_subscriber2->wait_for_subscription_sync;
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1");
+is($result, qq(0|itab1), 'initial data synced for itab1 on subscriber 1');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_1");
+is($result, qq(1|itab1), 'initial data synced for itab1_1 on subscriber 1');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_2");
+is($result, qq(2|itab1), 'initial data synced for itab1_2 on subscriber 1');
+
+$node_publisher->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+$node_subscriber2->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 DROP PUBLICATION dummy_pub");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION dummy_pub");
+
# Tests for replication using root table identity and schema
# publisher
@@ -877,4 +930,190 @@ $result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab5_1 ORDER BY 1");
is($result, qq(4||1), 'updates of tab5 replicated correctly');
+# Test that replication works for older inheritance/trigger setups as well.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab1_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab1_1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO itab1_2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab1_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab1_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (CHECK (a = 1)) INHERITS (itab2)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab2_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab2_1 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab2_trigger
+ BEFORE INSERT ON itab2
+ FOR EACH ROW EXECUTE FUNCTION itab2_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab2_1', 'itab2')");
+
+# itab2_1 should be published using its own identity here, since its parent is
+# not included. itab1_1 should be published via its parent, itab1, without
+# duplicating the rows.
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub_viaroot ADD TABLE itab1, itab1_1, itab2_1");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (2, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (0, 'itab2')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2')");
+
+# Subscriber 1 only subscribes to some of the partitions, and does not set up
+# partition triggers, to check for the correct routing.
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (a int, b text)");
+
+# Subscriber 2 has different partition names for itab1, and it doesn't partition
+# itab2 at all.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_subscriber2->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO public.itab1_part1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO public.itab1_part2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_subscriber2->safe_psql('postgres', "
+ CREATE TRIGGER itab_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab_trigger();");
+$node_subscriber2->safe_psql('postgres', "
+ ALTER TABLE itab1 ENABLE ALWAYS TRIGGER itab_trigger;");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot REFRESH PUBLICATION");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 REFRESH PUBLICATION");
+
+$node_subscriber1->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
+
+# check that data is synced correctly
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 1');
+
+# all of the data should have been routed to itab1 directly (there are no
+# triggers on subscriber 1 to move it elsewhere)
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data correctly routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2), 'initial data synced for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1), 'initial data correctly routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2), 'initial data synced for itab2 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1-new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2-new')");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+$node_publisher->wait_for_catchup('sub2');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2
+1|itab2-new), 'new data routed for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_part1 ORDER BY 1, 2");
+is($result, qq(1|itab1
+1|itab1-new), 'new data moved to itab1_part1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2
+1|itab2-new), 'new data routed for itab2 on subscriber 2');
+
done_testing();
--
2.25.1
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-03-07 10:40 ` Aleksander Alekseev <[email protected]>
2023-03-08 22:07 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2 siblings, 1 reply; 18+ messages in thread
From: Aleksander Alekseev @ 2023-03-07 10:40 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Jacob Champion <[email protected]>
Hi Jacob,
> I'm going to register this in CF for feedback.
Many thanks for the updated patch.
Despite the fact that the patch is still work in progress all in all
it looks very good to me.
So far I only have a couple of nitpicks, mostly regarding the code coverage [1]:
```
+ tablename = get_rel_name(tableoid);
+ if (tablename == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", tableoid)));
+ rootname = get_rel_name(rootoid);
+ if (rootname == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", rootoid)));
```
```
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(descRow), descRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch logical descendants for
table \"%s.%s\" from publisher: %s",
+ nspname, relname, res->err)));
```
```
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
+ pfree(cmd.data);
+ if (res->status != WALRCV_OK_COPY_OUT)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not start initial contents copy
for table \"%s.%s\" from remote %s: %s",
+ lrel.nspname, lrel.relname, quoted_name,
res->err)));
```
These new ereport() paths are never executed when we run the tests.
I'm not 100% sure if they are "should never happen in practice" cases
or not. If they are, I suggest adding corresponding comments.
Otherwise we have to test these paths.
```
+ else
+ {
+ /* For older servers, we only COPY the table itself. */
+ char *quoted = quote_qualified_identifier(lrel->nspname,
+ lrel->relname);
+ *to_copy = lappend(*to_copy, quoted);
+ }
```
Also we have to be extra careful with this code path because it is not
test-covered too.
```
+Datum
+pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
+{
+#define NUM_SYNC_TABLES_ELEM 1
```
What is this macro for?
```
+{ oid => '8137', descr => 'get list of tables to copy during initial sync',
+ proname => 'pg_get_publication_rels_to_sync', prorows => '10',
proretset => 't',
+ provolatile => 's', prorettype => 'regclass', proargtypes => 'regclass text',
+ proargnames => '{rootid,pubname}',
+ prosrc => 'pg_get_publication_rels_to_sync' },
```
Something seems odd here. Is there a chance that it can return
different results even within one statement, especially considering
the fact that pg_set_logical_root() is VOLATILE? Maybe
pg_get_publication_rels_to_sync() should be VOLATILE too [2].
```
+{ oid => '8136', descr => 'mark a table root for logical replication',
+ proname => 'pg_set_logical_root', provolatile => 'v', proparallel => 'u',
+ prorettype => 'void', proargtypes => 'regclass regclass',
+ prosrc => 'pg_set_logical_root' },
```
Shouldn't we also have pg_unset(reset?)_logical_root?
[1]: https://github.com/afiskon/pgscripts/blob/master/code-coverage.sh
[2]: https://www.postgresql.org/docs/current/xfunc-volatility.html
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-03-07 10:40 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
@ 2023-03-08 22:07 ` Jacob Champion <[email protected]>
0 siblings, 0 replies; 18+ messages in thread
From: Jacob Champion @ 2023-03-08 22:07 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Tue, Mar 7, 2023 at 2:40 AM Aleksander Alekseev
<[email protected]> wrote:
> So far I only have a couple of nitpicks, mostly regarding the code coverage [1]:
Yeah, I need to work on error cases and their coverage in general.
There are more cases that I need to reject as well (marked TODO).
> +Datum
> +pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
> +{
> +#define NUM_SYNC_TABLES_ELEM 1
> ```
>
> What is this macro for?
Whoops, that's cruft from an intermediate implementation. Will fix in
the next draft.
> +{ oid => '8137', descr => 'get list of tables to copy during initial sync',
> + proname => 'pg_get_publication_rels_to_sync', prorows => '10',
> proretset => 't',
> + provolatile => 's', prorettype => 'regclass', proargtypes => 'regclass text',
> + proargnames => '{rootid,pubname}',
> + prosrc => 'pg_get_publication_rels_to_sync' },
> ```
>
> Something seems odd here. Is there a chance that it can return
> different results even within one statement, especially considering
> the fact that pg_set_logical_root() is VOLATILE? Maybe
> pg_get_publication_rels_to_sync() should be VOLATILE too [2].
Hm. I'm not sure how this all should behave in the face of concurrent
structural changes, or how the existing publication queries handle
that same situation (e.g. partition attachment), so that's definitely
something for me to look into. At a glance, I'm not sure that
returning different results for the same table is more correct. And I
feel like a VOLATILE implementation might significantly impact the
JOIN/LATERAL performance in the pg_dump query? But I don't really know
how that's planned.
> +{ oid => '8136', descr => 'mark a table root for logical replication',
> + proname => 'pg_set_logical_root', provolatile => 'v', proparallel => 'u',
> + prorettype => 'void', proargtypes => 'regclass regclass',
> + prosrc => 'pg_set_logical_root' },
> ```
>
> Shouldn't we also have pg_unset(reset?)_logical_root?
My initial thought was that a one-way "upgrade" makes things easier to
reason about. But a one-way function is not good UX, so maybe we
should provide that. We'd need to verify and test what happens if you
undo/"detach" the logical tree during replication.
If it's okay to blindly replace any existing inhseqno with, say, 1 (on
a table with single inheritance), then we can reverse the process
safely. If not, we can't -- at least not with the current
implementation -- because we don't save the previous value anywhere.
Thanks for the review!
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-03-31 22:17 ` Peter Smith <[email protected]>
2023-03-31 23:35 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2 siblings, 1 reply; 18+ messages in thread
From: Peter Smith @ 2023-03-31 22:17 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Aleksander Alekseev <[email protected]>
On Wed, Mar 1, 2023 at 9:47 AM Jacob Champion <[email protected]> wrote:
>
> Hi,
>
> I'm going to register this in CF for feedback.
>
> Summary for potential reviewers: we don't use declarative partitions in
> the Timescale partitioning scheme, but it'd be really nice to be able to
> replicate between our tables and standard tables, or between two
> Timescale-partitioned tables with different layouts. This patch lets
> extensions (or savvy users) upgrade an existing inheritance relationship
> between two tables into a "logical partition" relationship, so that they
> can be handled with the publish_via_partition_root machinery.
>
> I hope this might also help pg_partman users migrate between old- and
> new-style partition schemes, but that's speculation.
>
OK, my understanding is that TimescaleDB uses some kind of
quasi-partitioned/inherited tables (aka hypertables? [1]) internally,
and your proposed WIP patch provides a set_logical_root() function
which combines with the logical replication (LR) PUBLICATION option
"publish_via_partition_root" to help to replicate those.
You also mentioned pg_partman. IIUC pg_partman is a partitioning
extension [2] that pre-dated the native PostgreSQL partitioning
introduced in PG10 (i.e. quite a while ago). I guess it would be a
very niche group of users that are still using pg_partman old-style
(pre-PG10) partitions and want to migrate them but have not already
done so. Also, the pg_partman README [3] says since v4.0.0 there is
extensive support for native PostgreSQL partitions, so perhaps
existing LR already works for those.
Outside the scope of special TimescaleDB tables and the speculated
pg_partman old-style table migration, will this proposed new feature
have any other application? In other words, do you know if this
proposal will be of any benefit to the *normal* users who just have
native PostgreSQL inherited tables they want to replicate? I haven’t
yet looked at the WIP patch TAP tests – so apologies for my question
if the benefits to normal users are self-evident from your test cases.
------
[1] https://docs.timescale.com/use-timescale/latest/hypertables/about-hypertables/
[2] https://www.crunchydata.com/blog/native-partitioning-with-postgres
[3] https://github.com/pgpartman/pg_partman
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-03-31 22:17 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
@ 2023-03-31 23:35 ` Jacob Champion <[email protected]>
2023-04-03 13:13 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-06-16 13:25 ` Re: RFC: logical publication via inheritance root? Amit Kapila <[email protected]>
0 siblings, 2 replies; 18+ messages in thread
From: Jacob Champion @ 2023-03-31 23:35 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Aleksander Alekseev <[email protected]>
On Fri, Mar 31, 2023 at 3:17 PM Peter Smith <[email protected]> wrote:
> OK, my understanding is that TimescaleDB uses some kind of
> quasi-partitioned/inherited tables (aka hypertables? [1]) internally,
> and your proposed WIP patch provides a set_logical_root() function
> which combines with the logical replication (LR) PUBLICATION option
> "publish_via_partition_root" to help to replicate those.
Correct!
> You also mentioned pg_partman. IIUC pg_partman is a partitioning
> extension [2] that pre-dated the native PostgreSQL partitioning
> introduced in PG10 (i.e. quite a while ago). I guess it would be a
> very niche group of users that are still using pg_partman old-style
> (pre-PG10) partitions and want to migrate them but have not already
> done so.
Yeah. I've got no evidence either way, unfortunately -- on the one
hand, surely people have been able to upgrade by now? And on the
other, implementation inertia seems to override most other engineering
goals...
Probably best to ask the partman users, and not me. :D Or assume it's
a non-benefit unless someone says otherwise (even then, the partman
maintainers would need to agree it's useful and add support for this).
> Outside the scope of special TimescaleDB tables and the speculated
> pg_partman old-style table migration, will this proposed new feature
> have any other application? In other words, do you know if this
> proposal will be of any benefit to the *normal* users who just have
> native PostgreSQL inherited tables they want to replicate?
I think it comes down to why an inheritance scheme was used. If it's
because you want to group rows into named, queryable subsets (e.g. the
"cities/capitals" example in the docs [1]), I don't think this has any
utility, because I assume you'd want to replicate your subsets as-is.
But if it's because you've implemented a partitioning scheme of your
own (the docs still list reasons you might want to [2], even today),
and all you ever really do is interact with the root table, I think
this feature will give you some of the same benefits that
publish_via_partition_root gives native partition users. We're very
much in that boat, but I don't know how many others are.
Thanks!
--Jacob
[1] https://www.postgresql.org/docs/15/tutorial-inheritance.html
[2] https://www.postgresql.org/docs/15/ddl-partitioning.html#DDL-PARTITIONING-USING-INHERITANCE
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-03-31 22:17 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
2023-03-31 23:35 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-04-03 13:13 ` Aleksander Alekseev <[email protected]>
1 sibling, 0 replies; 18+ messages in thread
From: Aleksander Alekseev @ 2023-04-03 13:13 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Jacob Champion <[email protected]>; Peter Smith <[email protected]>
Hi,
> > Outside the scope of special TimescaleDB tables and the speculated
> > pg_partman old-style table migration, will this proposed new feature
> > have any other application? In other words, do you know if this
> > proposal will be of any benefit to the *normal* users who just have
> > native PostgreSQL inherited tables they want to replicate?
>
> I think it comes down to why an inheritance scheme was used. If it's
> because you want to group rows into named, queryable subsets (e.g. the
> "cities/capitals" example in the docs [1]), I don't think this has any
> utility, because I assume you'd want to replicate your subsets as-is.
>
> But if it's because you've implemented a partitioning scheme of your
> own (the docs still list reasons you might want to [2], even today),
> and all you ever really do is interact with the root table, I think
> this feature will give you some of the same benefits that
> publish_via_partition_root gives native partition users. We're very
> much in that boat, but I don't know how many others are.
I would like to point out that inheritance is merely a tool for
modeling data. Its use cases are not limited to only partitioning,
although many people ended up using it for this purpose back when we
didn't have a proper built-in partitioning. So unless we are going to
remove inheritance in nearest releases (*) I believe it should work
with logical replication in a sane and convenient way.
Correct me if I'm wrong, but I got an impression that the patch tries
to accomplish just that.
(*) Which personally I believe would be a good change. Unlikely to
happen, though.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-03-31 22:17 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
2023-03-31 23:35 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-06-16 13:25 ` Amit Kapila <[email protected]>
1 sibling, 0 replies; 18+ messages in thread
From: Amit Kapila @ 2023-06-16 13:25 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: Peter Smith <[email protected]>; PostgreSQL Hackers <[email protected]>; Aleksander Alekseev <[email protected]>
On Sat, Apr 1, 2023 at 5:06 AM Jacob Champion <[email protected]> wrote:
>
> On Fri, Mar 31, 2023 at 3:17 PM Peter Smith <[email protected]> wrote:
>
> > Outside the scope of special TimescaleDB tables and the speculated
> > pg_partman old-style table migration, will this proposed new feature
> > have any other application? In other words, do you know if this
> > proposal will be of any benefit to the *normal* users who just have
> > native PostgreSQL inherited tables they want to replicate?
>
> I think it comes down to why an inheritance scheme was used. If it's
> because you want to group rows into named, queryable subsets (e.g. the
> "cities/capitals" example in the docs [1]), I don't think this has any
> utility, because I assume you'd want to replicate your subsets as-is.
>
I also think so and your idea to have a function like
pg_set_logical_root() seems to make the inheritance hierarchy behaves
as a declarative partitioning scheme for the purpose of logical
replication.
> But if it's because you've implemented a partitioning scheme of your
> own (the docs still list reasons you might want to [2], even today),
> and all you ever really do is interact with the root table, I think
> this feature will give you some of the same benefits that
> publish_via_partition_root gives native partition users. We're very
> much in that boat, but I don't know how many others are.
>
I agree that there may still be cases as pointed out by you where
people want to use inheritance as a mechanism for partitioning but I
feel those would still be in the minority. Personally, I am not very
excited about this idea.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-04-04 03:53 ` Peter Smith <[email protected]>
2023-04-04 15:14 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2 siblings, 1 reply; 18+ messages in thread
From: Peter Smith @ 2023-04-04 03:53 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Aleksander Alekseev <[email protected]>
FYI, the WIP patch does not seem to apply cleanly anymore using the latest HEAD.
See the cfbot rebase logs [1].
------
[1] http://cfbot.cputube.org/patch_42_4225.log
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-04-04 03:53 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
@ 2023-04-04 15:14 ` Jacob Champion <[email protected]>
2023-06-06 15:50 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2023-04-04 15:14 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>; Aleksander Alekseev <[email protected]>
On Mon, Apr 3, 2023 at 8:53 PM Peter Smith <[email protected]> wrote:
>
> FYI, the WIP patch does not seem to apply cleanly anymore using the latest HEAD.
Yes, sorry -- after 062a84442, the architecture needs to change in a
way that I'm still working through. I've moved the patch to Waiting on
Author while I figure out the rebase.
Thanks,
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-04-04 03:53 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
2023-04-04 15:14 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-06-06 15:50 ` Jacob Champion <[email protected]>
2023-06-29 23:46 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2023-06-06 15:50 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; Peter Smith <[email protected]>
On 4/4/23 08:14, Jacob Champion wrote:
> Yes, sorry -- after 062a84442, the architecture needs to change in a
> way that I'm still working through. I've moved the patch to Waiting on
> Author while I figure out the rebase.
Okay -- that took longer than I wanted, but here's a rebased patchset
that I'll call v2.
Commit 062a84442 necessitated some rework of the new
pg_get_publication_rels_to_sync() helper. It now takes a list of
publications so that we can handle conflicts in the pubviaroot settings.
This is more complicated than before -- unlike partitions, standard
inheritance trees can selectively publish tables that aren't leaves. But
I think I've finally settled on some semantics for it which are
unsurprising.
As part of that, I've pulled out a patch in 0001 which I hope is
independently useful. Today, there appears to be no way to check which
relid a table will be published through, short of creating a
subscription just to see what happens. 0001 introduces
pg_get_relation_publishing_info() to surface this information, which
makes testing it easier and also makes it possible to inspect what's
happening with more complicated publication setups.
0001 also moves the determination of publish_as_relid out of the
pgoutput plugin and into a pg_publication helper function, because
unless I've missed something crucial, it doesn't seem like an output
plugin is really free to make that decision independently of the
publication settings. The subscriber is not going to ask a plugin for
the right tables to COPY during initial sync, so the plugin had better
be using the same logic as the core.
Many TODOs and upthread points of feedback are still pending, and I
think that several of them are actually symptoms of one architectural
problem with my patch:
- The volatility classifications of pg_set_logical_root() and
pg_get_publication_rels_to_sync() appear to conflict
- A dump/restore cycle loses the new marker
- Inheritance can be tampered with after the logical root has been set
- There's currently no way to clear a logical root after setting it
I wonder if pg_set_logical_root() might be better implemented as part of
ALTER TABLE. Maybe with a relation option? If it all went through ALTER
TABLE ONLY ... SET, then we wouldn't have to worry about a user
modifying roots while reading pg_get_publication_rels_to_sync() in the
same query. The permissions checks should be more consistent with less
effort, and there's an existing way to set/clear the option that already
plays well with pg_dump and pg_upgrade. The downsides I can see are the
need to handle simultaneous changes to INHERIT and SET (since we'd be
manipulating pg_inherits in both), as well as the fact that ALTER TABLE
... SET defaults to altering the entire table hierarchy, which may be
bad UX for this case.
--Jacob
Attachments:
[text/x-patch] v2-0001-pgoutput-refactor-publication-cache-construction.patch (18.1K, ../../[email protected]/2-v2-0001-pgoutput-refactor-publication-cache-construction.patch)
download | inline diff:
From 761b91b3f173adffac9f73132c2f1c9fc8bcbec3 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Fri, 7 Apr 2023 09:55:27 -0700
Subject: [PATCH v2 1/2] pgoutput: refactor publication cache construction
Breaking this logic into a standalone helper should help expose the
behavior for testing. Additionally, move the implementation under the
pg_publication catalog helpers, since it seems like it's inherent to the
publication settings (and it must match the behavior of the initial
sync, which doesn't seem to be controlled by the output plugin at all).
---
src/backend/catalog/pg_publication.c | 215 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 137 +------------
src/include/catalog/pg_proc.dat | 8 +
src/include/catalog/pg_publication.h | 4 +
src/test/regress/expected/publication.out | 26 +++
src/test/regress/sql/publication.sql | 15 ++
6 files changed, 272 insertions(+), 133 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index c488b6370b..df9abf09c3 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1259,3 +1259,218 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+List *
+process_relation_publications(Oid relid, const List *publications,
+ PublicationActions *pubactions,
+ Oid *publish_as_relid)
+{
+ Oid schemaId = get_rel_namespace(relid);
+ List *pubids = GetRelationPublications(relid);
+
+ /*
+ * We don't acquire a lock on the namespace system table as we build
+ * the cache entry using a historic snapshot and all the later changes
+ * are absorbed while decoding WAL.
+ */
+ List *schemaPubids = GetSchemaPublications(schemaId);
+ ListCell *lc;
+ int publish_ancestor_level = 0;
+ bool am_partition = get_rel_relispartition(relid);
+ char relkind = get_rel_relkind(relid);
+ List *rel_publications = NIL;
+
+ *publish_as_relid = relid;
+
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ bool publish = false;
+
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications. Also
+ * track the ancestor level for this publication.
+ */
+ Oid pub_relid = relid;
+ int ancestor_level = 0;
+
+ /*
+ * If this is a FOR ALL TABLES publication, pick the partition
+ * root and set the ancestor level accordingly.
+ */
+ if (pub->alltables)
+ {
+ publish = true;
+ if (pub->pubviaroot && am_partition)
+ {
+ List *ancestors = get_partition_ancestors(relid);
+
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
+ }
+
+ if (!publish)
+ {
+ bool ancestor_published = false;
+
+ /*
+ * For a partition, check if any of the ancestors are
+ * published. If so, note down the topmost ancestor that is
+ * published via this publication, which will be used as the
+ * relation via which to publish the partition's changes.
+ */
+ if (am_partition)
+ {
+ Oid ancestor;
+ int level;
+ List *ancestors = get_partition_ancestors(relid);
+
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
+
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
+ {
+ pub_relid = ancestor;
+ ancestor_level = level;
+ }
+ }
+ }
+
+ if (list_member_oid(pubids, pub->oid) ||
+ list_member_oid(schemaPubids, pub->oid) ||
+ ancestor_published)
+ publish = true;
+ }
+
+ /*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
+ * Don't publish changes for partitioned tables, because
+ * publishing those of its partitions suffices, unless partition
+ * changes won't be published due to pubviaroot being set.
+ */
+ if (publish &&
+ (relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
+ {
+ pubactions->pubinsert |= pub->pubactions.pubinsert;
+ pubactions->pubupdate |= pub->pubactions.pubupdate;
+ pubactions->pubdelete |= pub->pubactions.pubdelete;
+ pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we need to check if the already
+ * calculated level is higher than the new one. If yes, we can
+ * ignore the new value (as it's a child). Otherwise the new
+ * value is an ancestor, so we keep it.
+ */
+ if (publish_ancestor_level > ancestor_level)
+ continue;
+
+ /*
+ * If we found an ancestor higher up in the tree, discard the
+ * list of publications through which we replicate it, and use
+ * the new ancestor.
+ */
+ if (publish_ancestor_level < ancestor_level)
+ {
+ *publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
+
+ /* reset the publication list for this relation */
+ rel_publications = NIL;
+ }
+ else
+ {
+ /* Same ancestor level, has to be the same OID. */
+ Assert(*publish_as_relid == pub_relid);
+ }
+
+ /* Track publications for this ancestor. */
+ rel_publications = lappend(rel_publications, pub);
+ }
+ }
+
+ list_free(pubids);
+ list_free(schemaPubids);
+
+ return rel_publications;
+}
+
+Datum
+pg_get_relation_publishing_info(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ List *publications = NIL;
+ ArrayType *arr;
+ Datum *elems;
+ int nelems,
+ i;
+ TupleDesc tupdesc;
+ HeapTuple htup;
+
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ arr = PG_GETARG_ARRAYTYPE_P(1);
+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT, &elems, NULL,
+ &nelems);
+
+ /* Get Oids of tables from each publication. */
+ for (i = 0; i < nelems; i++)
+ {
+ char *pubname = TextDatumGetCString(elems[i]);
+ Publication *pub = GetPublicationByName(pubname, false);
+
+ publications = lappend(publications, pub);
+ }
+
+ {
+ List *rel_publications;
+ PublicationActions pubactions;
+ Oid publish_as_relid;
+ ListCell *lc;
+ Datum *puboids;
+ ArrayType *puboidarray;
+ Datum values[6];
+ bool nulls[6];
+
+ rel_publications =
+ process_relation_publications(relid, publications, &pubactions,
+ &publish_as_relid);
+
+ /* Translate the rel_publications List into an OID array. */
+ puboids = palloc(sizeof(Datum) * list_length(rel_publications));
+
+ foreach(lc, rel_publications)
+ {
+ Publication *pub = lfirst(lc);
+
+ i = foreach_current_index(lc);
+ puboids[i] = ObjectIdGetDatum(pub->oid);
+ }
+
+ puboidarray = construct_array(puboids, list_length(rel_publications),
+ OIDOID, sizeof(Oid), true, TYPALIGN_INT);
+
+ values[0] = PointerGetDatum(puboidarray);
+ values[1] = ObjectIdGetDatum(publish_as_relid);
+ values[2] = BoolGetDatum(pubactions.pubinsert);
+ values[3] = BoolGetDatum(pubactions.pubupdate);
+ values[4] = BoolGetDatum(pubactions.pubdelete);
+ values[5] = BoolGetDatum(pubactions.pubtruncate);
+
+ memset(nulls, false, sizeof(nulls));
+
+ htup = heap_form_tuple(tupdesc, values, nulls);
+ }
+
+ PG_RETURN_DATUM(HeapTupleGetDatum(htup));
+}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index b08ca55041..640676e7dc 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1986,20 +1986,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Validate the entry */
if (!entry->replicate_valid)
{
- Oid schemaId = get_rel_namespace(relid);
- List *pubids = GetRelationPublications(relid);
-
- /*
- * We don't acquire a lock on the namespace system table as we build
- * the cache entry using a historic snapshot and all the later changes
- * are absorbed while decoding WAL.
- */
- List *schemaPubids = GetSchemaPublications(schemaId);
- ListCell *lc;
- Oid publish_as_relid = relid;
- int publish_ancestor_level = 0;
- bool am_partition = get_rel_relispartition(relid);
- char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
/* Reload publications if needed before use. */
@@ -2063,123 +2049,10 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
* but here we only need to consider ones that the subscriber
* requested.
*/
- foreach(lc, data->publications)
- {
- Publication *pub = lfirst(lc);
- bool publish = false;
-
- /*
- * Under what relid should we publish changes in this publication?
- * We'll use the top-most relid across all publications. Also
- * track the ancestor level for this publication.
- */
- Oid pub_relid = relid;
- int ancestor_level = 0;
-
- /*
- * If this is a FOR ALL TABLES publication, pick the partition
- * root and set the ancestor level accordingly.
- */
- if (pub->alltables)
- {
- publish = true;
- if (pub->pubviaroot && am_partition)
- {
- List *ancestors = get_partition_ancestors(relid);
-
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
- }
- }
-
- if (!publish)
- {
- bool ancestor_published = false;
-
- /*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
- * published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
- */
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
-
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
-
- if (ancestor != InvalidOid)
- {
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
- }
- }
-
- if (list_member_oid(pubids, pub->oid) ||
- list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
- publish = true;
- }
-
- /*
- * If the relation is to be published, determine actions to
- * publish, and list of columns, if appropriate.
- *
- * Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
- */
- if (publish &&
- (relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
- {
- entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
- entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
- entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
- entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
-
- /*
- * We want to publish the changes as the top-most ancestor
- * across all publications. So we need to check if the already
- * calculated level is higher than the new one. If yes, we can
- * ignore the new value (as it's a child). Otherwise the new
- * value is an ancestor, so we keep it.
- */
- if (publish_ancestor_level > ancestor_level)
- continue;
-
- /*
- * If we found an ancestor higher up in the tree, discard the
- * list of publications through which we replicate it, and use
- * the new ancestor.
- */
- if (publish_ancestor_level < ancestor_level)
- {
- publish_as_relid = pub_relid;
- publish_ancestor_level = ancestor_level;
-
- /* reset the publication list for this relation */
- rel_publications = NIL;
- }
- else
- {
- /* Same ancestor level, has to be the same OID. */
- Assert(publish_as_relid == pub_relid);
- }
-
- /* Track publications for this ancestor. */
- rel_publications = lappend(rel_publications, pub);
- }
- }
-
- entry->publish_as_relid = publish_as_relid;
+ rel_publications =
+ process_relation_publications(relid, data->publications,
+ &entry->pubactions,
+ &entry->publish_as_relid);
/*
* Initialize the tuple slot, map, and row filter. These are only used
@@ -2198,8 +2071,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
pgoutput_column_list_init(data, rel_publications, entry);
}
- list_free(pubids);
- list_free(schemaPubids);
list_free(rel_publications);
entry->replicate_valid = true;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6996073989..ed67168374 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11833,6 +11833,14 @@
proname => 'pg_relation_is_publishable', provolatile => 's',
prorettype => 'bool', proargtypes => 'regclass',
prosrc => 'pg_relation_is_publishable' },
+{ oid => '8139',
+ descr => 'get information on how a relation will be published via a list of publications',
+ proname => 'pg_get_relation_publishing_info', provariadic => 'text',
+ provolatile => 's', prorettype => 'record', proargtypes => 'regclass _text',
+ proallargtypes => '{regclass,_text,_oid,oid,bool,bool,bool,bool}',
+ proargmodes => '{i,v,o,o,o,o,o,o}',
+ proargnames => '{relid,pubnames,pubids,pubasrelid,pubinsert,pubupdate,pubdelete,pubtruncate}',
+ prosrc => 'pg_get_relation_publishing_info' },
# rls
{ oid => '3298',
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 6ecaa2a01e..d4e3488917 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -155,4 +155,8 @@ extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
MemoryContext mcxt);
+extern List *process_relation_publications(Oid relid, const List *publications,
+ PublicationActions *pubactions,
+ Oid *publish_as_relid);
+
#endif /* PG_PUBLICATION_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 69dc6cfd85..dd8fc99111 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -5,6 +5,17 @@ CREATE ROLE regress_publication_user LOGIN SUPERUSER;
CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_stream (VARIADIC pubnames text[])
+ RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
+ -- For each publication, show each published root alongside the tables which
+ -- are published via its OID.
+ SELECT p.pubname, rpi.pubasrelid::regclass, pr.prrelid::regclass
+ FROM pg_publication_rel pr
+ JOIN pg_publication p ON (p.oid = pr.prpubid),
+ pg_get_relation_publishing_info(pr.prrelid, VARIADIC pubnames) rpi
+ WHERE p.pubname = ANY (pubnames)
+ ORDER BY p.oid, 2, 3;
+$$ LANGUAGE sql;
-- suppress warning that depends on wal_level
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_default;
@@ -1686,6 +1697,13 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+SELECT * FROM published_stream('pub');
+ pubname | published | synced
+---------+-----------+-----------------
+ pub | sch1.tbl1 | sch1.tbl1
+ pub | sch1.tbl1 | sch2.tbl1_part1
+(2 rows)
+
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
@@ -1712,6 +1730,13 @@ SELECT * FROM pg_publication_tables;
pub | sch2 | tbl1_part1 | {a} |
(1 row)
+SELECT * FROM published_stream('pub');
+ pubname | published | synced
+---------+-----------------+-----------------
+ pub | sch1.tbl1 | sch1.tbl1
+ pub | sch2.tbl1_part1 | sch2.tbl1_part1
+(2 rows)
+
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
DROP TABLE sch1.tbl1;
@@ -1732,6 +1757,7 @@ DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
+DROP FUNCTION published_stream;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
DROP ROLE regress_publication_user_dummy;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index d5051a5e74..6bf9554d48 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -6,6 +6,18 @@ CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_stream (VARIADIC pubnames text[])
+ RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
+ -- For each publication, show each published root alongside the tables which
+ -- are published via its OID.
+ SELECT p.pubname, rpi.pubasrelid::regclass, pr.prrelid::regclass
+ FROM pg_publication_rel pr
+ JOIN pg_publication p ON (p.oid = pr.prpubid),
+ pg_get_relation_publishing_info(pr.prrelid, VARIADIC pubnames) rpi
+ WHERE p.pubname = ANY (pubnames)
+ ORDER BY p.oid, 2, 3;
+$$ LANGUAGE sql;
+
-- suppress warning that depends on wal_level
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_default;
@@ -1064,6 +1076,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
@@ -1078,6 +1091,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
@@ -1096,6 +1110,7 @@ DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
+DROP FUNCTION published_stream;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
--
2.25.1
[text/x-patch] v2-0002-WIP-introduce-pg_set_logical_root-for-use-with-pu.patch (70.6K, ../../[email protected]/3-v2-0002-WIP-introduce-pg_set_logical_root-for-use-with-pu.patch)
download | inline diff:
From 0397a525839e1dff3063c69872fac1161ff47c72 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Mon, 26 Sep 2022 13:23:51 -0700
Subject: [PATCH v2 2/2] WIP: introduce pg_set_logical_root for use with
pubviaroot
Allows regular inherited tables to be published via their root table,
just like partitions. This works by hijacking pg_inherit's inhseqno
column, and replacing a (single) existing entry for the child with the
value zero, indicating that it should be treated as a logical partition
by the publication machinery.
Initial sync works by asking the publisher for a list of logical
descendants of the published table, then COPYing them one-by-one into
the root. The publisher reuses the existing pubviaroot logic, adding the
new logical roots to code that previously looked only for partition
roots.
Known bugs/TODOs:
- The pg_inherits machinery doesn't prohibit changes to inheritance
after an entry has been marked as a logical root.
- I haven't given any thought to interactions with row filters, or to
column lists.
- I'm not sure that I'm taking all the necessary locks yet, and those I
do take may be taken in the wrong order.
- Dump and upgrade aren't supported yet.
---
src/backend/catalog/pg_inherits.c | 212 ++++++++++++-
src/backend/catalog/pg_publication.c | 282 +++++++++++++-----
src/backend/commands/publicationcmds.c | 10 +
src/backend/partitioning/partdesc.c | 3 +-
src/backend/replication/logical/tablesync.c | 257 ++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 1 -
src/include/catalog/pg_inherits.h | 7 +-
src/include/catalog/pg_proc.dat | 12 +
src/test/regress/expected/publication.out | 266 +++++++++++++++++
src/test/regress/sql/publication.sql | 121 ++++++++
src/test/subscription/t/013_partition.pl | 312 ++++++++++++++++++++
11 files changed, 1309 insertions(+), 174 deletions(-)
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index da969bd2f9..b59f93c833 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -24,14 +24,22 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_inherits.h"
+#include "catalog/partition.h"
+#include "miscadmin.h"
#include "parser/parse_type.h"
#include "storage/lmgr.h"
+#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/fmgrprotos.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+static List * find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only);
+
/*
* Entry of a hash table used in find_all_inheritors. See below.
*/
@@ -59,14 +67,14 @@ List *
find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
{
return find_inheritance_children_extended(parentrelId, true, lockmode,
- NULL, NULL);
+ NULL, NULL, false);
}
/*
* find_inheritance_children_extended
*
* As find_inheritance_children, with more options regarding detached
- * partitions.
+ * partitions and logical roots.
*
* If a partition's pg_inherits row is marked "detach pending",
* *detached_exist (if not null) is set true.
@@ -78,16 +86,21 @@ find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
* whether the transaction that marked those partitions as detached appears
* committed to the active snapshot. In addition, *detached_xmin (if not null)
* is set to the xmin of the row of the detached partition.
+ *
+ * If logical_only is true, only tables marked explicitly via
+ * pg_set_logical_root() are included in the output list.
*/
List *
find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
LOCKMODE lockmode, bool *detached_exist,
- TransactionId *detached_xmin)
+ TransactionId *detached_xmin,
+ bool logical_only)
{
List *list = NIL;
Relation relation;
+ Oid index;
SysScanDesc scan;
- ScanKeyData key[1];
+ ScanKeyData key[2];
HeapTuple inheritsTuple;
Oid inhrelid;
Oid *oidarr;
@@ -110,14 +123,24 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
numoids = 0;
relation = table_open(InheritsRelationId, AccessShareLock);
+ index = InheritsParentIndexId;
ScanKeyInit(&key[0],
Anum_pg_inherits_inhparent,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(parentrelId));
- scan = systable_beginscan(relation, InheritsParentIndexId, true,
- NULL, 1, key);
+ if (logical_only)
+ {
+ index = InheritsParentSeqnoIndexId;
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+ }
+
+ scan = systable_beginscan(relation, index, true,
+ NULL, logical_only ? 2 : 1, key);
while ((inheritsTuple = systable_getnext(scan)) != NULL)
{
@@ -254,6 +277,20 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
*/
List *
find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
+{
+ return find_all_inheritors_internal(parentrelId, lockmode, numparents,
+ false);
+}
+
+List *
+find_all_logical_inheritors(Oid parentrelId)
+{
+ return find_all_inheritors_internal(parentrelId, NoLock, NULL, true);
+}
+
+static List *
+find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only)
{
/* hash table for O(1) rel_oid -> rel_numparents cell lookup */
HTAB *seen_rels;
@@ -290,7 +327,9 @@ find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
ListCell *lc;
/* Get the direct children of this rel */
- currentchildren = find_inheritance_children(currentrel, lockmode);
+ currentchildren =
+ find_inheritance_children_extended(currentrel, true, lockmode,
+ NULL, NULL, logical_only);
/*
* Add to the queue only those children not already seen. This avoids
@@ -655,3 +694,162 @@ PartitionHasPendingDetach(Oid partoid)
elog(ERROR, "relation %u is not a partition", partoid);
return false; /* keep compiler quiet */
}
+
+static Oid
+get_logical_parent_worker(Relation inhRel, Oid relid)
+{
+ SysScanDesc scan;
+ ScanKeyData key[2];
+ Oid result = InvalidOid;
+ HeapTuple tuple;
+
+ ScanKeyInit(&key[0],
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 2, key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ Form_pg_inherits form = (Form_pg_inherits) GETSTRUCT(tuple);
+ result = form->inhparent;
+ }
+
+ systable_endscan(scan);
+
+ return result;
+}
+
+static void
+get_logical_ancestors_worker(Relation inhRel, Oid relid, List **ancestors)
+{
+ Oid parentOid;
+
+ /*
+ * Recursion ends at the topmost level, ie., when there's no parent.
+ */
+ parentOid = get_logical_parent_worker(inhRel, relid);
+ if (parentOid == InvalidOid)
+ return;
+
+ *ancestors = lappend_oid(*ancestors, parentOid);
+ get_logical_ancestors_worker(inhRel, parentOid, ancestors);
+}
+
+List *
+get_logical_ancestors(Oid relid, bool is_partition)
+{
+ List *result = NIL;
+ Relation inhRel;
+
+ /* For partitions, this is identical to get_partition_ancestors(). */
+ if (is_partition)
+ return get_partition_ancestors(relid);
+
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
+ get_logical_ancestors_worker(inhRel, relid, &result);
+ table_close(inhRel, AccessShareLock);
+
+ return result;
+}
+
+bool
+has_logical_parent(Relation inhRel, Oid relid)
+{
+ return (get_logical_parent_worker(inhRel, relid) != InvalidOid);
+}
+
+Datum
+pg_set_logical_root(PG_FUNCTION_ARGS)
+{
+ Oid tableoid = PG_GETARG_OID(0);
+ Oid rootoid = PG_GETARG_OID(1);
+ char *tablename;
+ char *rootname;
+ Relation inhRel;
+ ScanKeyData key;
+ SysScanDesc scan;
+ Oid parent = InvalidOid;
+ HeapTuple tuple, copyTuple;
+ Form_pg_inherits form;
+
+ /*
+ * Check that the tables exist.
+ * TODO: check identical schemas too? or does replication handle that?
+ */
+ tablename = get_rel_name(tableoid);
+ if (tablename == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", tableoid)));
+ rootname = get_rel_name(rootoid);
+ if (rootname == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("OID %u does not refer to a table", rootoid)));
+
+ /* Check ownership. */
+ if (!object_ownercheck(RelationRelationId, tableoid, GetUserId()))
+ aclcheck_error(ACLCHECK_NOT_OWNER,
+ get_relkind_objtype(get_rel_relkind(tableoid)),
+ tablename);
+ if (!object_ownercheck(RelationRelationId, rootoid, GetUserId()))
+ aclcheck_error(ACLCHECK_NOT_OWNER,
+ get_relkind_objtype(get_rel_relkind(rootoid)),
+ rootname);
+
+ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
+ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
+
+ /*
+ * We have to make sure that the inheritance relationship already exists,
+ * and that there is only one existing parent for this table.
+ *
+ * TODO: do we have to lock the tables themselves to avoid races?
+ */
+ ScanKeyInit(&key,
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(tableoid));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 1, &key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ form = (Form_pg_inherits) GETSTRUCT(tuple);
+ parent = form->inhparent;
+ copyTuple = heap_copytuple(tuple);
+
+ if (HeapTupleIsValid(systable_getnext(scan)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" inherits from multiple tables",
+ tablename)));
+ }
+
+ if (parent != rootoid)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" does not inherit from intended root table \"%s\"",
+ tablename, rootname)));
+
+ systable_endscan(scan);
+
+ /* Mark the inheritance as a logical root by setting it to zero. */
+ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
+ form->inhseqno = 0;
+
+ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
+
+ heap_freetuple(copyTuple);
+ table_close(inhRel, RowExclusiveLock);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index df9abf09c3..f23c75d058 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -51,6 +51,7 @@ typedef struct
Oid relid; /* OID of published table */
Oid pubid; /* OID of publication that publishes this
* table. */
+ bool viaroot; /* does pubid use publish_via_partition_root? */
} published_rel;
static void publication_translate_columns(Relation targetrel, List *columns,
@@ -180,56 +181,46 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
}
/*
- * Returns true if the ancestor is in the list of published relations.
- * Otherwise, returns false.
- */
-static bool
-is_ancestor_member_tableinfos(Oid ancestor, List *table_infos)
-{
- ListCell *lc;
-
- foreach(lc, table_infos)
- {
- Oid relid = ((published_rel *) lfirst(lc))->relid;
-
- if (relid == ancestor)
- return true;
- }
-
- return false;
-}
-
-/*
- * Filter out the partitions whose parent tables are also present in the list.
+ * Filter out the tables who are already being published via a logical root.
+ *
+ * If we only had partitioned tables to check, this would be much easier: it's
+ * impossible for a partition root to be in this list unless pubviaroot=true for
+ * that table, so we would just have to check for the existence of an ancestor,
+ * and if any were found then we'd filter the table.
+ *
+ * Logical roots add another wrinkle, because it's acceptable for a
+ * pubviaroot=false publication to contain a logical root. (Logical roots can
+ * contain data of their own, unlike partition roots.) It's also possible for a
+ * logical root to be included in a pubviaroot publication without any or all of
+ * its children. This is not possible for a partition root, where all of the
+ * leaves are implicitly included.
+ *
+ * So, to keep things relatively consistent, this now uses the same code that
+ * the output plugin uses to decide which relation OID to publish data under. If
+ * the publishing OID is not the same as the relation OID, we remove it from
+ * this list.
+ *
+ * XXX And that's expensive.
*/
static void
-filter_partitions(List *table_infos)
+filter_published_descendants(List *table_infos, List *publications)
{
ListCell *lc;
foreach(lc, table_infos)
{
- bool skip = false;
- List *ancestors = NIL;
- ListCell *lc2;
published_rel *table_info = (published_rel *) lfirst(lc);
+ Oid publish_as_relid;
+ List *rel_publications;
- if (get_rel_relispartition(table_info->relid))
- ancestors = get_partition_ancestors(table_info->relid);
-
- foreach(lc2, ancestors)
- {
- Oid ancestor = lfirst_oid(lc2);
-
- if (is_ancestor_member_tableinfos(ancestor, table_infos))
- {
- skip = true;
- break;
- }
- }
+ rel_publications =
+ process_relation_publications(table_info->relid, publications, NULL,
+ &publish_as_relid);
- if (skip)
+ if (publish_as_relid != table_info->relid)
table_infos = foreach_delete_current(table_infos, lc);
+
+ list_free(rel_publications);
}
}
@@ -805,11 +796,15 @@ List *
GetAllTablesPublicationRelations(bool pubviaroot)
{
Relation classRel;
+ Relation inhRel;
ScanKeyData key[1];
TableScanDesc scan;
HeapTuple tuple;
List *result = NIL;
+ /* TODO: is there a required order to acquire these locks? */
+ if (pubviaroot)
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
classRel = table_open(RelationRelationId, AccessShareLock);
ScanKeyInit(&key[0],
@@ -825,7 +820,8 @@ GetAllTablesPublicationRelations(bool pubviaroot)
Oid relid = relForm->oid;
if (is_publishable_class(relid, relForm) &&
- !(relForm->relispartition && pubviaroot))
+ !(relForm->relispartition && pubviaroot) &&
+ !(pubviaroot && has_logical_parent(inhRel, relid)))
result = lappend_oid(result, relid);
}
@@ -854,6 +850,9 @@ GetAllTablesPublicationRelations(bool pubviaroot)
}
table_close(classRel, AccessShareLock);
+ if (pubviaroot)
+ table_close(inhRel, AccessShareLock);
+
return result;
}
@@ -1070,6 +1069,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
int nelems,
i;
bool viaroot = false;
+ List *publications = NIL;
/* create a function context for cross-call persistence */
funcctx = SRF_FIRSTCALL_INIT();
@@ -1093,6 +1093,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
ListCell *lc;
pub_elem = GetPublicationByName(TextDatumGetCString(elems[i]), false);
+ publications = lappend(publications, pub_elem);
/*
* Publications support partitioned tables. If
@@ -1132,6 +1133,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
table_info->relid = lfirst_oid(lc);
table_info->pubid = pub_elem->oid;
+ table_info->viaroot = pub_elem->pubviaroot;
table_infos = lappend(table_infos, table_info);
}
@@ -1141,15 +1143,14 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
}
/*
- * If the publication publishes partition changes via their respective
- * root partitioned tables, we must exclude partitions in favor of
- * including the root partitioned tables. Otherwise, the function
- * could return both the child and parent tables which could cause
- * data of the child table to be double-published on the subscriber
- * side.
+ * If the publication publishes table changes via their respective
+ * logical root tables, we must exclude partitions in favor of
+ * including the root tables. Otherwise, the function could return
+ * both the child and parent tables which could cause data of the
+ * child table to be double-published on the subscriber side.
*/
if (viaroot)
- filter_partitions(table_infos);
+ filter_published_descendants(table_infos, publications);
/* Construct a tuple descriptor for the result rows. */
tupdesc = CreateTemplateTupleDesc(NUM_PUBLICATION_TABLES_ELEM);
@@ -1165,6 +1166,8 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
funcctx->tuple_desc = BlessTupleDesc(tupdesc);
funcctx->user_fctx = (void *) table_infos;
+ list_free(publications);
+
MemoryContextSwitchTo(oldcontext);
}
@@ -1260,6 +1263,118 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+/*
+ * Returns a list of tables that should be copied during initial sync for the
+ * given root table and publications.
+ *
+ * The only time this list consists of anything more than the table itself is
+ * when a publication's publish_via_partition_root is set to true and the table
+ * has inherited child tables in the publications that have been marked with
+ * pg_set_logical_root().
+ */
+Datum
+pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ List *tables = NIL;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ Oid rootid = PG_GETARG_OID(0);
+ MemoryContext oldcontext;
+ ArrayType *arr;
+ Datum *elems;
+ int nelems,
+ i;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ /*
+ * Deconstruct the parameter into elements where each element is a
+ * publication name.
+ */
+ arr = PG_GETARG_ARRAYTYPE_P(1);
+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT,
+ &elems, NULL, &nelems);
+
+ /* Get Oids of tables from each publication. */
+ for (i = 0; i < nelems; i++)
+ {
+ char *pubname = TextDatumGetCString(elems[i]);
+ Publication *publication = GetPublicationByName(pubname, false);
+ List *pub_tables = NIL;
+
+ /* TODO: do the tables in this list need to be locked? */
+ if (publication->pubviaroot)
+ pub_tables = find_all_logical_inheritors(rootid);
+ else
+ pub_tables = list_make1_oid(rootid);
+
+ if (!publication->alltables)
+ {
+ List *relids;
+ List *schemarelids;
+ List *published;
+ ListCell *cell;
+
+ relids = GetPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_ALL);
+ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_LEAF);
+ published = list_concat_unique_oid(relids, schemarelids);
+
+ /*
+ * First we have to check to make sure the root table is
+ * actually part of the publication; if not, none of its
+ * descendants' contents belong to it.
+ */
+ if (!list_member_oid(published, rootid))
+ continue;
+
+ /*
+ * Now filter out any descendants that aren't part of this
+ * particular publication.
+ */
+ foreach(cell, pub_tables)
+ {
+ Oid current = lfirst_oid(cell);
+
+ if (!list_member_oid(published, current))
+ pub_tables = foreach_delete_current(pub_tables, cell);
+ }
+ }
+
+ tables = list_concat_unique_oid(tables, pub_tables);
+ }
+
+ funcctx->user_fctx = (void *) tables;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+ tables = (List *) funcctx->user_fctx;
+
+ if (funcctx->call_cntr < list_length(tables))
+ {
+ Oid relid = list_nth_oid(tables, funcctx->call_cntr);
+
+ SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ }
+
+ SRF_RETURN_DONE(funcctx);
+}
+
List *
process_relation_publications(Oid relid, const List *publications,
PublicationActions *pubactions,
@@ -1279,8 +1394,7 @@ process_relation_publications(Oid relid, const List *publications,
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
-
- *publish_as_relid = relid;
+ Oid publish_candidate = relid;
foreach(lc, publications)
{
@@ -1296,55 +1410,57 @@ process_relation_publications(Oid relid, const List *publications,
int ancestor_level = 0;
/*
- * If this is a FOR ALL TABLES publication, pick the partition
+ * If this is a FOR ALL TABLES publication, pick the logical
* root and set the ancestor level accordingly.
*/
if (pub->alltables)
{
publish = true;
- if (pub->pubviaroot && am_partition)
+ if (pub->pubviaroot)
{
- List *ancestors = get_partition_ancestors(relid);
+ List *ancestors;
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ if (ancestors != NIL)
+ {
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
}
if (!publish)
{
bool ancestor_published = false;
+ Oid ancestor;
+ int level;
+ List *ancestors;
/*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
+ * Check if any of the logical ancestors (that is, partition
+ * parents or tables marked with pg_set_logical_root()) are
+ * published. If so, note down the topmost ancestor that is
* published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
+ * relation via which to publish this table's changes.
*/
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
-
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
- if (ancestor != InvalidOid)
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
{
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
+ pub_relid = ancestor;
+ ancestor_level = level;
}
}
if (list_member_oid(pubids, pub->oid) ||
list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
+ (am_partition && ancestor_published))
publish = true;
}
@@ -1359,10 +1475,13 @@ process_relation_publications(Oid relid, const List *publications,
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
- pubactions->pubinsert |= pub->pubactions.pubinsert;
- pubactions->pubupdate |= pub->pubactions.pubupdate;
- pubactions->pubdelete |= pub->pubactions.pubdelete;
- pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+ if (pubactions)
+ {
+ pubactions->pubinsert |= pub->pubactions.pubinsert;
+ pubactions->pubupdate |= pub->pubactions.pubupdate;
+ pubactions->pubdelete |= pub->pubactions.pubdelete;
+ pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+ }
/*
* We want to publish the changes as the top-most ancestor
@@ -1381,7 +1500,7 @@ process_relation_publications(Oid relid, const List *publications,
*/
if (publish_ancestor_level < ancestor_level)
{
- *publish_as_relid = pub_relid;
+ publish_candidate = pub_relid;
publish_ancestor_level = ancestor_level;
/* reset the publication list for this relation */
@@ -1390,7 +1509,7 @@ process_relation_publications(Oid relid, const List *publications,
else
{
/* Same ancestor level, has to be the same OID. */
- Assert(*publish_as_relid == pub_relid);
+ Assert(publish_candidate == pub_relid);
}
/* Track publications for this ancestor. */
@@ -1401,6 +1520,9 @@ process_relation_publications(Oid relid, const List *publications,
list_free(pubids);
list_free(schemaPubids);
+ if (publish_as_relid)
+ *publish_as_relid = publish_candidate;
+
return rel_publications;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f4ba572697..7e23bed6c0 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -238,6 +238,8 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* parent table, but the bitmap contains the replica identity
* information of the child table. So, get the column number of the
* child table as parent and child column order could be different.
+ *
+ * TODO: is this applicable to pg_set_logical_root()?
*/
if (context->pubviaroot)
{
@@ -286,6 +288,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
*
* Note that even though the row filter used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
+ *
+ * TODO: is this applicable to pg_set_logical_root()?
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
@@ -336,6 +340,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
* the column list.
*
* Returns true if any replica identity column is not covered by column list.
+ *
+ * TODO: pg_set_logical_root()?
*/
bool
pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
@@ -628,6 +634,8 @@ TransformPubWhereClauses(List *tables, const char *queryString,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
+ * TODO: decide how this interacts with pg_set_logical_root
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ -715,6 +723,8 @@ CheckPubRelationColumnList(char *pubname, List *tables,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
+ * TODO: decide if this interacts with pg_set_logical_root()
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
diff --git a/src/backend/partitioning/partdesc.c b/src/backend/partitioning/partdesc.c
index 7a2b5e57ff..dfb27990f9 100644
--- a/src/backend/partitioning/partdesc.c
+++ b/src/backend/partitioning/partdesc.c
@@ -162,7 +162,8 @@ RelationBuildPartitionDesc(Relation rel, bool omit_detached)
inhoids = find_inheritance_children_extended(RelationGetRelid(rel),
omit_detached, NoLock,
&detached_exist,
- &detached_xmin);
+ &detached_xmin,
+ false);
nparts = list_length(inhoids);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index c56d42dcd2..db0f56b05e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -757,11 +757,12 @@ copy_read_data(void *outbuf, int minread, int maxread)
/*
* Get information about remote relation in similar fashion the RELATION
* message provides during replication. This function also returns the relation
- * qualifications to be used in the COPY command.
+ * qualifications to be used in the COPY command, and the list of tables to COPY
+ * (which for most tables will contain only one entry).
*/
static void
fetch_remote_table_info(char *nspname, char *relname,
- LogicalRepRelation *lrel, List **qual)
+ LogicalRepRelation *lrel, List **qual, List **to_copy)
{
WalRcvExecResult *res;
StringInfoData cmd;
@@ -1072,6 +1073,87 @@ fetch_remote_table_info(char *nspname, char *relname,
walrcv_clear_result(res);
}
+ /*
+ * See if there are any other tables to be copied besides the original. This
+ * happens when a descendant in the inheritance relationship is marked with
+ * pg_set_logical_root() and is part of a publication with
+ * publish_via_partition_root = true.
+ *
+ * FIXME: if two publications have conflicting settings for
+ * publish_via_partition_root, this behaves strangely. It looks like that's
+ * true for regular partitions too...
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000)
+ {
+ Oid descRow[] = {TEXTOID, TEXTOID};
+ StringInfoData pub_names;
+
+ /* Build the pubname list. */
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ char *pubname = strVal(lfirst(lc));
+
+ if (foreach_current_index(lc) > 0)
+ appendStringInfoString(&pub_names, ", ");
+
+ appendStringInfoString(&pub_names, quote_literal_cstr(pubname));
+ }
+
+ /*
+ * Ask the server what it wants us to sync.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT n.nspname, c.relname"
+ " FROM pg_catalog.pg_publication p,"
+ " pg_catalog.pg_class c"
+ " JOIN pg_catalog.pg_namespace n"
+ " ON (c.relnamespace = n.oid)"
+ " JOIN LATERAL pg_catalog.pg_get_publication_rels_to_sync(%u::regclass, p.pubname) relid"
+ " ON (c.oid = relid)"
+ " WHERE p.pubname IN ( %s )",
+ lrel->remoteid, pub_names.data);
+
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(descRow), descRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch logical descendants for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *desc_nspname;
+ char *desc_relname;
+ char *quoted;
+
+ desc_nspname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+ desc_relname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ quoted = quote_qualified_identifier(desc_nspname, desc_relname);
+ *to_copy = lappend(*to_copy, quoted);
+
+ ExecClearTuple(slot);
+ }
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+
+ pfree(pub_names.data);
+ }
+ else
+ {
+ /* For older servers, we only COPY the table itself. */
+ char *quoted = quote_qualified_identifier(lrel->nspname,
+ lrel->relname);
+ *to_copy = lappend(*to_copy, quoted);
+ }
+
pfree(cmd.data);
}
@@ -1086,6 +1168,8 @@ copy_table(Relation rel)
LogicalRepRelMapEntry *relmapentry;
LogicalRepRelation lrel;
List *qual = NIL;
+ List *to_copy = NIL;
+ ListCell *cur;
WalRcvExecResult *res;
StringInfoData cmd;
CopyFromState cstate;
@@ -1095,7 +1179,8 @@ copy_table(Relation rel)
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &lrel, &qual,
+ &to_copy);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1104,105 +1189,109 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
- /* Start copy on the publisher. */
- initStringInfo(&cmd);
-
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ foreach(cur, to_copy)
{
- appendStringInfo(&cmd, "COPY %s (",
- quote_qualified_identifier(lrel.nspname, lrel.relname));
-
- /*
- * XXX Do we need to list the columns in all cases? Maybe we're
- * replicating all columns?
- */
- for (int i = 0; i < lrel.natts; i++)
- {
- if (i > 0)
- appendStringInfoString(&cmd, ", ");
+ char *quoted_name = lfirst(cur);
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- }
+ /* Start copy on the publisher. */
+ initStringInfo(&cmd);
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
- else
- {
- /*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
- */
- appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ /* Regular table with no row filter */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
- appendStringInfoString(&cmd, ", ");
- }
+ appendStringInfo(&cmd, "COPY %s (", quoted_name);
- appendStringInfoString(&cmd, " FROM ");
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're
+ * replicating all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
- /*
- * For regular tables, make sure we don't copy data from a child that
- * inherits the named table as those will be copied separately.
- */
- if (lrel.relkind == RELKIND_RELATION)
- appendStringInfoString(&cmd, "ONLY ");
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
- appendStringInfoString(&cmd, quote_qualified_identifier(lrel.nspname, lrel.relname));
- /* list of OR'ed filters */
- if (qual != NIL)
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
+ else
{
- ListCell *lc;
- char *q = strVal(linitial(qual));
+ /*
+ * For non-tables and tables with row filters, we need to do COPY
+ * (SELECT ...), but we can't just do SELECT * because we need to not
+ * copy generated columns. For tables with any row filters, build a
+ * SELECT query with OR'ed row filters for COPY.
+ */
+ appendStringInfoString(&cmd, "COPY (SELECT ");
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+
+ appendStringInfoString(&cmd, " FROM ");
+
+ /*
+ * For regular tables, make sure we don't copy data from a child that
+ * inherits the named table as those will be copied separately.
+ */
+ if (lrel.relkind == RELKIND_RELATION)
+ appendStringInfoString(&cmd, "ONLY ");
- appendStringInfo(&cmd, " WHERE %s", q);
- for_each_from(lc, qual, 1)
+ appendStringInfoString(&cmd, quoted_name);
+ /* list of OR'ed filters */
+ if (qual != NIL)
{
- q = strVal(lfirst(lc));
- appendStringInfo(&cmd, " OR %s", q);
+ ListCell *lc;
+ char *q = strVal(linitial(qual));
+
+ appendStringInfo(&cmd, " WHERE %s", q);
+ for_each_from(lc, qual, 1)
+ {
+ q = strVal(lfirst(lc));
+ appendStringInfo(&cmd, " OR %s", q);
+ }
+ list_free_deep(qual);
}
- list_free_deep(qual);
- }
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
- /*
- * Prior to v16, initial table synchronization will use text format even
- * if the binary option is enabled for a subscription.
- */
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000 &&
- MySubscription->binary)
- {
- appendStringInfoString(&cmd, " WITH (FORMAT binary)");
- options = list_make1(makeDefElem("format",
- (Node *) makeString("binary"), -1));
- }
+ /*
+ * Prior to v16, initial table synchronization will use text format even
+ * if the binary option is enabled for a subscription.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000 &&
+ MySubscription->binary)
+ {
+ appendStringInfoString(&cmd, " WITH (FORMAT binary)");
+ options = list_make1(makeDefElem("format",
+ (Node *) makeString("binary"), -1));
+ }
- res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
- pfree(cmd.data);
- if (res->status != WALRCV_OK_COPY_OUT)
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("could not start initial contents copy for table \"%s.%s\": %s",
- lrel.nspname, lrel.relname, res->err)));
- walrcv_clear_result(res);
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
+ pfree(cmd.data);
+ if (res->status != WALRCV_OK_COPY_OUT)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not start initial contents copy for table \"%s.%s\" from remote %s: %s",
+ lrel.nspname, lrel.relname, quoted_name, res->err)));
+ walrcv_clear_result(res);
- copybuf = makeStringInfo();
+ copybuf = makeStringInfo();
- pstate = make_parsestate(NULL);
- (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
- NULL, false, false);
+ pstate = make_parsestate(NULL);
+ (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
+ NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
- cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
+ attnamelist = make_copy_attnamelist(relmapentry);
+ cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
- /* Do the copy */
- (void) CopyFrom(cstate);
+ /* Do the copy */
+ (void) CopyFrom(cstate);
+ }
logicalrep_rel_close(relmapentry, NoLock);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 640676e7dc..7c54b46ac7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1462,7 +1462,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Switch relation if publishing via root. */
if (relentry->publish_as_relid != RelationGetRelid(relation))
{
- Assert(relation->rd_rel->relispartition);
ancestor = RelationIdGetRelation(relentry->publish_as_relid);
targetrel = ancestor;
}
diff --git a/src/include/catalog/pg_inherits.h b/src/include/catalog/pg_inherits.h
index ce154ab943..cfcacacf76 100644
--- a/src/include/catalog/pg_inherits.h
+++ b/src/include/catalog/pg_inherits.h
@@ -46,14 +46,17 @@ typedef FormData_pg_inherits *Form_pg_inherits;
DECLARE_UNIQUE_INDEX_PKEY(pg_inherits_relid_seqno_index, 2680, InheritsRelidSeqnoIndexId, on pg_inherits using btree(inhrelid oid_ops, inhseqno int4_ops));
DECLARE_INDEX(pg_inherits_parent_index, 2187, InheritsParentIndexId, on pg_inherits using btree(inhparent oid_ops));
+DECLARE_INDEX(pg_inherits_parent_seqno_index, 8138, InheritsParentSeqnoIndexId, on pg_inherits using btree(inhparent oid_ops, inhseqno int4_ops));
extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);
extern List *find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
- LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin);
+ LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin,
+ bool logical_only);
extern List *find_all_inheritors(Oid parentrelId, LOCKMODE lockmode,
List **numparents);
+extern List *find_all_logical_inheritors(Oid parentrelId);
extern bool has_subclass(Oid relationId);
extern bool has_superclass(Oid relationId);
extern bool typeInheritsFrom(Oid subclassTypeId, Oid superclassTypeId);
@@ -63,5 +66,7 @@ extern bool DeleteInheritsTuple(Oid inhrelid, Oid inhparent,
bool expect_detach_pending,
const char *childname);
extern bool PartitionHasPendingDetach(Oid partoid);
+extern List *get_logical_ancestors(Oid relid, bool is_partition);
+extern bool has_logical_parent(Relation inhRel, Oid relid);
#endif /* PG_INHERITS_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ed67168374..1dfc8fdd1e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11833,6 +11833,13 @@
proname => 'pg_relation_is_publishable', provolatile => 's',
prorettype => 'bool', proargtypes => 'regclass',
prosrc => 'pg_relation_is_publishable' },
+{ oid => '8137',
+ descr => 'get the relations to copy from all specified publications during a table\'s initial sync',
+ proname => 'pg_get_publication_rels_to_sync', prorows => '10',
+ provariadic => 'text', proretset => 't', provolatile => 's',
+ prorettype => 'regclass', proargtypes => 'regclass _text',
+ proargmodes => '{i,v}', proargnames => '{rootid,pubnames}',
+ prosrc => 'pg_get_publication_rels_to_sync' },
{ oid => '8139',
descr => 'get information on how a relation will be published via a list of publications',
proname => 'pg_get_relation_publishing_info', provariadic => 'text',
@@ -11999,6 +12006,11 @@
proname => 'pg_partition_root', prorettype => 'regclass',
proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+{ oid => '8136', descr => 'mark a table root for logical replication',
+ proname => 'pg_set_logical_root', provolatile => 'v', proparallel => 'u',
+ prorettype => 'void', proargtypes => 'regclass regclass',
+ prosrc => 'pg_set_logical_root' },
+
{ oid => '4350', descr => 'Unicode normalization',
proname => 'normalize', prorettype => 'text', proargtypes => 'text text',
prosrc => 'unicode_normalize_func' },
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index dd8fc99111..9587674edc 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -5,6 +5,17 @@ CREATE ROLE regress_publication_user LOGIN SUPERUSER;
CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_sync (VARIADIC pubnames text[])
+ RETURNS TABLE (published regclass, synced regclass) AS $$
+ -- Show each published table alongside the tables to be copied into it during
+ -- initial sync.
+ SELECT t.published, synced
+ FROM (SELECT DISTINCT relid::regclass AS published
+ FROM pg_get_publication_tables(VARIADIC pubnames)) t
+ JOIN LATERAL pg_get_publication_rels_to_sync(t.published, VARIADIC pubnames) synced
+ ON true
+ ORDER BY 1, 2;
+$$ LANGUAGE sql;
CREATE FUNCTION published_stream (VARIADIC pubnames text[])
RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
-- For each publication, show each published root alongside the tables which
@@ -1697,6 +1708,12 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+SELECT * FROM published_sync('pub');
+ published | synced
+-----------+-----------
+ sch1.tbl1 | sch1.tbl1
+(1 row)
+
SELECT * FROM published_stream('pub');
pubname | published | synced
---------+-----------+-----------------
@@ -1730,6 +1747,12 @@ SELECT * FROM pg_publication_tables;
pub | sch2 | tbl1_part1 | {a} |
(1 row)
+SELECT * FROM published_sync('pub');
+ published | synced
+-----------------+-----------------
+ sch2.tbl1_part1 | sch2.tbl1_part1
+(1 row)
+
SELECT * FROM published_stream('pub');
pubname | published | synced
---------+-----------------+-----------------
@@ -1752,12 +1775,255 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+-- Sanity check cases for pg_set_logical_root().
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+-- marking roots between unrelated tables is not allowed
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+ERROR: table "ipart1" does not inherit from intended root table "iroot"
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
+ERROR: table "ipart2" does not inherit from intended root table "tbl1"
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+ERROR: table "ipart2" inherits from multiple tables
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
+SELECT pg_set_logical_root('part', 'root'); -- should fail
+ERROR: must be owner of table part
+RESET SESSION AUTHORIZATION;
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
+SELECT pg_set_logical_root('part', 'root'); -- should also fail
+ERROR: must be owner of table root
+RESET SESSION AUTHORIZATION;
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+SELECT pg_set_logical_root('sch1.ipart1_a', 'sch1.ipart1');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+SELECT pg_set_logical_root('sch1.ipart1_a1', 'sch1.ipart1_a');
+ pg_set_logical_root
+---------------------
+
+(1 row)
+
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
+ WITH (publish_via_partition_root);
+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
+ ONLY sch1.ipart2
+ WITH (publish_via_partition_root);
+-- At this point, the published trees look like this:
+--
+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
+-- ------------------ ----------------------- -----------------------
+-- iroot iroot
+-- +- ipart1 ipart1 |
+-- | +- ipart1_a | +--- ipart1_a
+-- | +- ipart1_a1 +--- ipart1_a1 |
+-- +- ipart2 +- ipart2
+-- What a subscription to only ipub_root should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
+ relid
+----------------
+ sch1.iroot
+ sch1.ipart1
+ sch1.ipart2
+ sch1.ipart1_a
+ sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_sync('ipub_root');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a | sch1.ipart1_a
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_root');
+ pubname | published | synced
+-----------+----------------+----------------
+ ipub_root | sch1.iroot | sch1.iroot
+ ipub_root | sch1.ipart1 | sch1.ipart1
+ ipub_root | sch1.ipart2 | sch1.ipart2
+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
+ ipub_root | sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+-- What a subscription to only ipub_part1 should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
+ relid
+-------------
+ sch1.ipart1
+(1 row)
+
+SELECT * FROM published_sync('ipub_part1');
+ published | synced
+-------------+----------------
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+(2 rows)
+
+SELECT * FROM published_stream('ipub_part1');
+ pubname | published | synced
+------------+-------------+----------------
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+(2 rows)
+
+-- What a subscription to both ipub_root and ipub_part1 should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+------------+---------------
+ ipub_root | sch1.iroot
+ ipub_root | sch1.ipart1
+ ipub_root | sch1.ipart2
+ ipub_root | sch1.ipart1_a
+ ipub_part1 | sch1.ipart1
+(5 rows)
+
+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
+ published | synced
+---------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a | sch1.ipart1_a
+(5 rows)
+
+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
+ pubname | published | synced
+------------+---------------+----------------
+ ipub_root | sch1.iroot | sch1.iroot
+ ipub_root | sch1.ipart1 | sch1.ipart1
+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
+ ipub_root | sch1.ipart2 | sch1.ipart2
+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+(7 rows)
+
+-- What a subscription to both ipub_part1 and ipub_other should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+------------+-------------
+ ipub_part1 | sch1.ipart1
+ ipub_other | sch1.iroot
+(2 rows)
+
+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
+ published | synced
+-------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.iroot | sch1.ipart2
+ sch1.iroot | sch1.ipart1_a
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
+ pubname | published | synced
+------------+-------------+----------------
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+ ipub_other | sch1.iroot | sch1.iroot
+ ipub_other | sch1.iroot | sch1.ipart2
+ ipub_other | sch1.iroot | sch1.ipart1_a
+(5 rows)
+
+-- What a subscription to all three should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+------------+-------------
+ ipub_root | sch1.iroot
+ ipub_root | sch1.ipart1
+ ipub_part1 | sch1.ipart1
+ ipub_other | sch1.iroot
+(4 rows)
+
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+ published | synced
+-------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.iroot | sch1.ipart2
+ sch1.iroot | sch1.ipart1_a
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+ pubname | published | synced
+------------+-------------+----------------
+ ipub_root | sch1.iroot | sch1.iroot
+ ipub_root | sch1.iroot | sch1.ipart2
+ ipub_root | sch1.iroot | sch1.ipart1_a
+ ipub_root | sch1.ipart1 | sch1.ipart1
+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+ ipub_other | sch1.iroot | sch1.iroot
+ ipub_other | sch1.iroot | sch1.ipart2
+ ipub_other | sch1.iroot | sch1.ipart1_a
+(10 rows)
+
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1_a1;
+DROP TABLE sch1.ipart1_a;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
DROP FUNCTION published_stream;
+DROP FUNCTION published_sync;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
DROP ROLE regress_publication_user_dummy;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 6bf9554d48..a498c53862 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -6,6 +6,18 @@ CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_sync (VARIADIC pubnames text[])
+ RETURNS TABLE (published regclass, synced regclass) AS $$
+ -- Show each published table alongside the tables to be copied into it during
+ -- initial sync.
+ SELECT t.published, synced
+ FROM (SELECT DISTINCT relid::regclass AS published
+ FROM pg_get_publication_tables(VARIADIC pubnames)) t
+ JOIN LATERAL pg_get_publication_rels_to_sync(t.published, VARIADIC pubnames) synced
+ ON true
+ ORDER BY 1, 2;
+$$ LANGUAGE sql;
+
CREATE FUNCTION published_stream (VARIADIC pubnames text[])
RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
-- For each publication, show each published root alongside the tables which
@@ -1076,6 +1088,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_sync('pub');
SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
@@ -1091,6 +1104,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_sync('pub');
SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
@@ -1105,12 +1119,119 @@ ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
+-- Sanity check cases for pg_set_logical_root().
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+
+-- marking roots between unrelated tables is not allowed
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
+
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
+
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
+SELECT pg_set_logical_root('part', 'root'); -- should fail
+RESET SESSION AUTHORIZATION;
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
+SELECT pg_set_logical_root('part', 'root'); -- should also fail
+RESET SESSION AUTHORIZATION;
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+
+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
+
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+
+SELECT pg_set_logical_root('sch1.ipart1_a', 'sch1.ipart1');
+SELECT pg_set_logical_root('sch1.ipart1_a1', 'sch1.ipart1_a');
+
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
+ WITH (publish_via_partition_root);
+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
+ ONLY sch1.ipart2
+ WITH (publish_via_partition_root);
+
+-- At this point, the published trees look like this:
+--
+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
+-- ------------------ ----------------------- -----------------------
+-- iroot iroot
+-- +- ipart1 ipart1 |
+-- | +- ipart1_a | +--- ipart1_a
+-- | +- ipart1_a1 +--- ipart1_a1 |
+-- +- ipart2 +- ipart2
+
+-- What a subscription to only ipub_root should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
+SELECT * FROM published_sync('ipub_root');
+SELECT * FROM published_stream('ipub_root');
+
+-- What a subscription to only ipub_part1 should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
+SELECT * FROM published_sync('ipub_part1');
+SELECT * FROM published_stream('ipub_part1');
+
+-- What a subscription to both ipub_root and ipub_part1 should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
+
+-- What a subscription to both ipub_part1 and ipub_other should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
+
+-- What a subscription to all three should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
+
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1_a1;
+DROP TABLE sch1.ipart1_a;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
DROP FUNCTION published_stream;
+DROP FUNCTION published_sync;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 275fb3b525..bc40c68030 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -388,6 +388,59 @@ $node_subscriber1->append_conf('postgresql.conf',
"log_min_messages = warning");
$node_subscriber1->reload;
+# Make sure standard inheritance setups aren't broken by the new
+# pg_set_logical_root() handling.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (LIKE itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (LIKE itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_2 VALUES (2, 'itab1')");
+
+# Regression: Create a publication for an unrelated table and set
+# publish_via_partition_root. This should have no effect at all, since itab1
+# isn't supposed to be using this publication...
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION dummy_pub FOR TABLE tab1 WITH (publish_via_partition_root = true)");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 ADD PUBLICATION dummy_pub");
+$node_subscriber2->wait_for_subscription_sync;
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1");
+is($result, qq(0|itab1), 'initial data synced for itab1 on subscriber 2');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_1");
+is($result, qq(1|itab1), 'initial data synced for itab1_1 on subscriber 2');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_2");
+is($result, qq(2|itab1), 'initial data synced for itab1_2 on subscriber 2');
+
+$node_publisher->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+$node_subscriber2->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 DROP PUBLICATION dummy_pub");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION dummy_pub");
+
# Tests for replication using root table identity and schema
# publisher
@@ -886,4 +939,263 @@ $result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab5_1 ORDER BY 1");
is($result, qq(4||1), 'updates of tab5 replicated correctly');
+# Test that replication works for older inheritance/trigger setups as well.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab1_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab1_1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO itab1_2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab1_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab1_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (CHECK (a = 1)) INHERITS (itab2)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab2_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab2_1 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab2_trigger
+ BEFORE INSERT ON itab2
+ FOR EACH ROW EXECUTE FUNCTION itab2_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab2_1', 'itab2')");
+
+# itab2_1 should be published using its own identity here, since its parent is
+# not included. itab1_1 should be published via its parent, itab1, without
+# duplicating the rows.
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub_viaroot ADD TABLE itab1, itab1_1, itab2_1");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (2, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (0, 'itab2')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2')");
+
+# Subscriber 1 only subscribes to some of the partitions, and does not set up
+# partition triggers, to check for the correct routing.
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (a int, b text)");
+
+# Subscriber 2 has different partition names for itab1, and it doesn't partition
+# itab2 at all.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_subscriber2->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO public.itab1_part1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO public.itab1_part2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_subscriber2->safe_psql('postgres', "
+ CREATE TRIGGER itab_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab_trigger();");
+$node_subscriber2->safe_psql('postgres', "
+ ALTER TABLE itab1 ENABLE ALWAYS TRIGGER itab_trigger;");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot REFRESH PUBLICATION");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 REFRESH PUBLICATION");
+
+$node_subscriber1->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
+
+# check that data is synced correctly
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 1');
+
+# all of the data should have been routed to itab1 directly (there are no
+# triggers on subscriber 1 to move it elsewhere)
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data correctly routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2), 'initial data synced for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1), 'initial data correctly routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2), 'initial data synced for itab2 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1-new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2-new')");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+$node_publisher->wait_for_catchup('sub2');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2
+1|itab2-new), 'new data routed for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_part1 ORDER BY 1, 2");
+is($result, qq(1|itab1
+1|itab1-new), 'new data moved to itab1_part1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2
+1|itab2-new), 'new data routed for itab2 on subscriber 2');
+
+# Finally, check the effect of mixed publish_via_partition_root settings on
+# logical roots.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3_1 () INHERITS (itab3)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab3_1', 'itab3')");
+$node_publisher->safe_psql('postgres',
+ "SELECT pg_set_logical_root('itab3_1_1', 'itab3_1')");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3 VALUES (1, 'itab3')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3_1 VALUES (2, 'itab3_1')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3_1_1 VALUES (3, 'itab3_1_1')");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION ipub3 FOR TABLE itab3 *");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION ipub3_1 FOR TABLE itab3_1 * WITH (publish_via_partition_root = true)");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3_1 () INHERITS (itab3)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION mixed CONNECTION '$publisher_connstr' PUBLICATION ipub3, ipub3_1");
+$node_subscriber2->wait_for_subscription_sync;
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3");
+is($result, qq(1|itab3), 'initial data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1), 'initial data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
+is($result, qq(), 'initial data routed for itab3_1_1 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3 VALUES (4, 'itab3-new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1 VALUES (4, 'itab3_1-new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1_1 VALUES (4, 'itab3_1_1-new')");
+
+$node_publisher->wait_for_catchup('mixed');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3 ORDER BY 1, 2");
+is($result, qq(1|itab3
+4|itab3-new), 'new data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1
+4|itab3_1-new
+4|itab3_1_1-new), 'new data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
+is($result, qq(), 'new data routed for itab3_1_1 on subscriber 2');
+
done_testing();
--
2.25.1
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-04-04 03:53 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
2023-04-04 15:14 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-06-06 15:50 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-06-29 23:46 ` Jacob Champion <[email protected]>
2023-07-19 11:54 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2023-06-29 23:46 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>
On 6/6/23 08:50, Jacob Champion wrote:
> Commit 062a84442 necessitated some rework of the new
> pg_get_publication_rels_to_sync() helper. It now takes a list of
> publications so that we can handle conflicts in the pubviaroot settings.
> This is more complicated than before -- unlike partitions, standard
> inheritance trees can selectively publish tables that aren't leaves. But
> I think I've finally settled on some semantics for it which are
> unsurprising.
The semantics I've picked are wrong :( I've accidentally reintroduced a
bug that's similar to the one discussed upthread, where if you have two
leaf tables published through different roots:
tree pub1 pub2
----- ----- -----
A - A
B C B - - -
D D D
then a subscription on both publications will duplicate the data in the
leaf (D in the above example). I need to choose semantics that are
closer to the current behavior of partitions.
> I wonder if pg_set_logical_root() might be better implemented as part of
> ALTER TABLE. Maybe with a relation option? If it all went through ALTER
> TABLE ONLY ... SET, then we wouldn't have to worry about a user
> modifying roots while reading pg_get_publication_rels_to_sync() in the
> same query. The permissions checks should be more consistent with less
> effort, and there's an existing way to set/clear the option that already
> plays well with pg_dump and pg_upgrade.
I've implemented ALTER TABLE in v3; I like it a lot more. The new
reloption is named publish_via_parent. So now we can unset the flag and
dump/restore.
v3 also fixes a nasty uninitialized stack variable, along with a bad
collation assumption I made.
> The downsides I can see are the
> need to handle simultaneous changes to INHERIT and SET (since we'd be
> manipulating pg_inherits in both),
(This didn't turn out to be as bad as I feared.)
> as well as the fact that ALTER TABLE
> ... SET defaults to altering the entire table hierarchy, which may be
> bad UX for this case.
This was just wrong -- ALTER TABLE ... SET does not recurse. I think the
docs are misleading here, and I'm not sure why we allow an explicit '*'
after a table name if we're going to ignore it. But I also don't really
want to poke at that in this patchset, if I can avoid it.
--Jacob
1: 761b91b3f1 = 1: 7e68f96af6 pgoutput: refactor publication cache construction
2: 0397a52583 ! 2: a48563919e WIP: introduce pg_set_logical_root for use with pubviaroot
@@ Metadata
Author: Jacob Champion <[email protected]>
## Commit message ##
- WIP: introduce pg_set_logical_root for use with pubviaroot
+ WIP: introduce publish_via_parent for logical replication
- Allows regular inherited tables to be published via their root table,
- just like partitions. This works by hijacking pg_inherit's inhseqno
- column, and replacing a (single) existing entry for the child with the
- value zero, indicating that it should be treated as a logical partition
- by the publication machinery.
+ This new relation option allows regular inherited tables to be published
+ via their root table, just like partitions. This works by hijacking
+ pg_inherit's inhseqno column, and replacing a (single) existing entry
+ for the child with the value zero, indicating that it should be treated
+ as a logical partition by the publication machinery.
Initial sync works by asking the publisher for a list of logical
descendants of the published table, then COPYing them one-by-one into
@@ Commit message
roots.
Known bugs/TODOs:
- - The pg_inherits machinery doesn't prohibit changes to inheritance
- after an entry has been marked as a logical root.
+ - If two publications publish the same leaves of a table hierarchy, but
+ have different roots, the shared leaf tables will be incorrectly
+ duplicated when subscribing to both publications simultaneously.
- I haven't given any thought to interactions with row filters, or to
column lists.
- I'm not sure that I'm taking all the necessary locks yet, and those I
do take may be taken in the wrong order.
- - Dump and upgrade aren't supported yet.
+
+ ## src/backend/access/common/reloptions.c ##
+@@ src/backend/access/common/reloptions.c: static relopt_bool boolRelOpts[] =
+ },
+ false
+ },
++ {
++ {
++ "publish_via_parent",
++ "Replicate this table's contents via its parent table when publishing with publish_via_partition_root",
++ RELOPT_KIND_HEAP,
++ AccessExclusiveLock
++ },
++ false
++ },
+ {
+ {
+ "fastupdate",
+@@ src/backend/access/common/reloptions.c: default_reloptions(Datum reloptions, bool validate, relopt_kind kind)
+ offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, analyze_scale_factor)},
+ {"user_catalog_table", RELOPT_TYPE_BOOL,
+ offsetof(StdRdOptions, user_catalog_table)},
++ {"publish_via_parent", RELOPT_TYPE_BOOL,
++ offsetof(StdRdOptions, publish_via_parent)},
+ {"parallel_workers", RELOPT_TYPE_INT,
+ offsetof(StdRdOptions, parallel_workers)},
+ {"vacuum_index_cleanup", RELOPT_TYPE_ENUM,
## src/backend/catalog/pg_inherits.c ##
@@
@@ src/backend/catalog/pg_inherits.c: find_inheritance_children(Oid parentrelId, LO
* committed to the active snapshot. In addition, *detached_xmin (if not null)
* is set to the xmin of the row of the detached partition.
+ *
-+ * If logical_only is true, only tables marked explicitly via
-+ * pg_set_logical_root() are included in the output list.
++ * If logical_only is true, only tables marked explicitly via publish_via_parent
++ * are included in the output list.
*/
List *
find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
@@ src/backend/catalog/pg_inherits.c: PartitionHasPendingDetach(Oid partoid)
+has_logical_parent(Relation inhRel, Oid relid)
+{
+ return (get_logical_parent_worker(inhRel, relid) != InvalidOid);
-+}
-+
-+Datum
-+pg_set_logical_root(PG_FUNCTION_ARGS)
-+{
-+ Oid tableoid = PG_GETARG_OID(0);
-+ Oid rootoid = PG_GETARG_OID(1);
-+ char *tablename;
-+ char *rootname;
-+ Relation inhRel;
-+ ScanKeyData key;
-+ SysScanDesc scan;
-+ Oid parent = InvalidOid;
-+ HeapTuple tuple, copyTuple;
-+ Form_pg_inherits form;
-+
-+ /*
-+ * Check that the tables exist.
-+ * TODO: check identical schemas too? or does replication handle that?
-+ */
-+ tablename = get_rel_name(tableoid);
-+ if (tablename == NULL)
-+ ereport(ERROR,
-+ (errcode(ERRCODE_UNDEFINED_TABLE),
-+ errmsg("OID %u does not refer to a table", tableoid)));
-+ rootname = get_rel_name(rootoid);
-+ if (rootname == NULL)
-+ ereport(ERROR,
-+ (errcode(ERRCODE_UNDEFINED_TABLE),
-+ errmsg("OID %u does not refer to a table", rootoid)));
-+
-+ /* Check ownership. */
-+ if (!object_ownercheck(RelationRelationId, tableoid, GetUserId()))
-+ aclcheck_error(ACLCHECK_NOT_OWNER,
-+ get_relkind_objtype(get_rel_relkind(tableoid)),
-+ tablename);
-+ if (!object_ownercheck(RelationRelationId, rootoid, GetUserId()))
-+ aclcheck_error(ACLCHECK_NOT_OWNER,
-+ get_relkind_objtype(get_rel_relkind(rootoid)),
-+ rootname);
-+
-+ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
-+ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
-+
-+ /*
-+ * We have to make sure that the inheritance relationship already exists,
-+ * and that there is only one existing parent for this table.
-+ *
-+ * TODO: do we have to lock the tables themselves to avoid races?
-+ */
-+ ScanKeyInit(&key,
-+ Anum_pg_inherits_inhrelid,
-+ BTEqualStrategyNumber, F_OIDEQ,
-+ ObjectIdGetDatum(tableoid));
-+
-+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
-+ NULL, 1, &key);
-+ tuple = systable_getnext(scan);
-+ if (HeapTupleIsValid(tuple))
-+ {
-+ form = (Form_pg_inherits) GETSTRUCT(tuple);
-+ parent = form->inhparent;
-+ copyTuple = heap_copytuple(tuple);
-+
-+ if (HeapTupleIsValid(systable_getnext(scan)))
-+ ereport(ERROR,
-+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
-+ errmsg("table \"%s\" inherits from multiple tables",
-+ tablename)));
-+ }
-+
-+ if (parent != rootoid)
-+ ereport(ERROR,
-+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
-+ errmsg("table \"%s\" does not inherit from intended root table \"%s\"",
-+ tablename, rootname)));
-+
-+ systable_endscan(scan);
-+
-+ /* Mark the inheritance as a logical root by setting it to zero. */
-+ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
-+ form->inhseqno = 0;
-+
-+ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
-+
-+ heap_freetuple(copyTuple);
-+ table_close(inhRel, RowExclusiveLock);
-+
-+ PG_RETURN_VOID();
+}
## src/backend/catalog/pg_publication.c ##
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
+ * The only time this list consists of anything more than the table itself is
+ * when a publication's publish_via_partition_root is set to true and the table
+ * has inherited child tables in the publications that have been marked with
-+ * pg_set_logical_root().
++ * publish_via_parent.
+ */
+Datum
+pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
+ * Check if any of the logical ancestors (that is, partition
-+ * parents or tables marked with pg_set_logical_root()) are
++ * parents or tables marked with publish_via_parent) are
+ * published. If so, note down the topmost ancestor that is
* published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
return rel_publications;
}
+@@ src/backend/catalog/pg_publication.c: pg_get_relation_publishing_info(PG_FUNCTION_ARGS)
+
+ {
+ List *rel_publications;
+- PublicationActions pubactions;
++ PublicationActions pubactions = {0};
+ Oid publish_as_relid;
+ ListCell *lc;
+ Datum *puboids;
## src/backend/commands/publicationcmds.c ##
@@ src/backend/commands/publicationcmds.c: contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
@@ src/backend/commands/publicationcmds.c: contain_invalid_rfcolumn_walker(Node *no
* information of the child table. So, get the column number of the
* child table as parent and child column order could be different.
+ *
-+ * TODO: is this applicable to pg_set_logical_root()?
++ * TODO: is this applicable to publish_via_parent?
*/
if (context->pubviaroot)
{
@@ src/backend/commands/publicationcmds.c: pub_rf_contains_invalid_column(Oid pubid
* Note that even though the row filter used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
+ *
-+ * TODO: is this applicable to pg_set_logical_root()?
++ * TODO: is this applicable to publish_via_parent?
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
@@ src/backend/commands/publicationcmds.c: pub_rf_contains_invalid_column(Oid pubid
*
* Returns true if any replica identity column is not covered by column list.
+ *
-+ * TODO: pg_set_logical_root()?
++ * TODO: publish_via_parent?
*/
bool
pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
@@ src/backend/commands/publicationcmds.c: TransformPubWhereClauses(List *tables, c
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
-+ * TODO: decide how this interacts with pg_set_logical_root
++ * TODO: decide how this interacts with publish_via_parent
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ src/backend/commands/publicationcmds.c: CheckPubRelationColumnList(char *pubname
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
-+ * TODO: decide if this interacts with pg_set_logical_root()
++ * TODO: decide if this interacts with publish_via_parent
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ## src/backend/commands/tablecmds.c ##
+@@ src/backend/commands/tablecmds.c: static bool MergeCheckConstraint(List *constraints, char *name, Node *expr);
+ static void MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel);
+ static void MergeConstraintsIntoExisting(Relation child_rel, Relation parent_rel);
+ static void StoreCatalogInheritance(Oid relationId, List *supers,
+- bool child_is_partition);
++ bool child_is_partition,
++ bool publish_via_parent);
+ static void StoreCatalogInheritance1(Oid relationId, Oid parentOid,
+ int32 seqNumber, Relation inhRelation,
+ bool child_is_partition);
+@@ src/backend/commands/tablecmds.c: static void ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel,
+ const char *tablespacename, LOCKMODE lockmode);
+ static void ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode);
+ static void ATExecSetTableSpaceNoStorage(Relation rel, Oid newTableSpace);
++static void change_publish_via_parent(Relation rel, bool set);
+ static void ATExecSetRelOptions(Relation rel, List *defList,
+ AlterTableType operation,
+ LOCKMODE lockmode);
+@@ src/backend/commands/tablecmds.c: DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
+ }
+
+ /* Store inheritance information for new rel. */
+- StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL);
++ StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL,
++ RelationIsPublishedViaParent(rel));
+
+ /*
+ * Process the partitioning specification (if any) and store the partition
+@@ src/backend/commands/tablecmds.c: MergeCheckConstraint(List *constraints, char *name, Node *expr)
+ */
+ static void
+ StoreCatalogInheritance(Oid relationId, List *supers,
+- bool child_is_partition)
++ bool child_is_partition, bool publish_via_parent)
+ {
+ Relation relation;
+ int32 seqNumber;
+@@ src/backend/commands/tablecmds.c: StoreCatalogInheritance(Oid relationId, List *supers,
+ */
+ Assert(OidIsValid(relationId));
+
++ /*
++ * publish_via_parent requires exactly one parent. Try to keep this
++ * messaging consistent with ALTER TABLE/change_publish_via_parent().
++ *
++ * Earlier, in MergeAttributes, we already checked that we had ownership of
++ * the parent table, so we don't need to check that again the way that
++ * change_publish_via_parent() does.
++ */
++ if (publish_via_parent && list_length(supers) != 1)
++ ereport(ERROR,
++ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
++ errmsg((list_length(supers) == 0)
++ ? "table \"%s\" does not inherit from any tables"
++ : "table \"%s\" inherits from multiple tables",
++ get_rel_name(relationId))));
++
+ if (supers == NIL)
+ return;
+
+@@ src/backend/commands/tablecmds.c: StoreCatalogInheritance(Oid relationId, List *supers,
+ */
+ relation = table_open(InheritsRelationId, RowExclusiveLock);
+
+- seqNumber = 1;
++ /*
++ * Normally inhseqno starts at one, but for publish_via_parent, it's given a
++ * sentinel value of zero. (In that case, we'll only go through this loop
++ * once, as the list_length() check above enforces.)
++ */
++ seqNumber = publish_via_parent ? 0 : 1;
+ foreach(entry, supers)
+ {
+ Oid parentOid = lfirst_oid(entry);
+@@ src/backend/commands/tablecmds.c: ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel, const char *tablespacen
+ tab->newTableSpace = tablespaceId;
+ }
+
++static void
++change_publish_via_parent(Relation rel, bool set)
++{
++ Relation inhRel;
++ ScanKeyData key;
++ SysScanDesc scan;
++ Oid parentOid;
++ Relation parentRel;
++ HeapTuple tuple, copyTuple;
++ Form_pg_inherits form;
++
++ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
++ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
++
++ /*
++ * We have to make sure that an inheritance relationship already exists, and
++ * that there is only one candidate parent for this table.
++ *
++ * TODO: do we have to lock the tables themselves to avoid races?
++ */
++ ScanKeyInit(&key,
++ Anum_pg_inherits_inhrelid,
++ BTEqualStrategyNumber, F_OIDEQ,
++ ObjectIdGetDatum(RelationGetRelid(rel)));
++
++ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
++ NULL, 1, &key);
++ tuple = systable_getnext(scan);
++
++ if (!HeapTupleIsValid(tuple))
++ ereport(ERROR,
++ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
++ errmsg("table \"%s\" does not inherit from any tables",
++ RelationGetRelationName(rel))));
++
++ form = (Form_pg_inherits) GETSTRUCT(tuple);
++ parentOid = form->inhparent;
++ copyTuple = heap_copytuple(tuple);
++
++ if (HeapTupleIsValid(systable_getnext(scan)))
++ ereport(ERROR,
++ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
++ errmsg("table \"%s\" inherits from multiple tables",
++ RelationGetRelationName(rel))));
++
++ systable_endscan(scan);
++
++ if (set)
++ {
++ /*
++ * Open our parent table and check permissions. Like ATTACH PARTITION,
++ * we require ownership of the parent table in addition to the child
++ * (which has already been checked in ATPrepCmd).
++ *
++ * TODO: what lock strength is needed here? Is nesting it inside the
++ * inhRel lock a risk?
++ */
++ parentRel = table_open(parentOid, AccessExclusiveLock);
++ ATSimplePermissions(AT_SetRelOptions, parentRel, ATT_TABLE);
++ table_close(parentRel, NoLock); /* TODO: do we need to hang onto the lock? */
++ }
++ else
++ {
++ /*
++ * Ownership of the parent isn't needed for unsetting the flag, just
++ * like we wouldn't check ownership during NO INHERIT.
++ */
++ }
++
++ /*
++ * Mark the inheritance as a logical root by setting inhseqno to zero.
++ * Unmark it by setting inhseqno to one (since there's only one parent).
++ */
++ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
++ form->inhseqno = set ? 0 : 1;
++
++ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
++
++ heap_freetuple(copyTuple);
++ table_close(inhRel, RowExclusiveLock);
++}
++
+ /*
+ * Set, reset, or replace reloptions.
+ */
+@@ src/backend/commands/tablecmds.c: ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation,
+ }
+ }
+
++ /* Extra work for publish_via_parent */
++ if (rel->rd_rel->relkind == RELKIND_RELATION)
++ {
++ List *heap_options = untransformRelOptions(newOptions);
++ ListCell *cell;
++ bool curSetting = RelationIsPublishedViaParent(rel);
++ DefElem *pubElem = NULL;
++
++ foreach (cell, heap_options)
++ {
++ DefElem *defel = lfirst_node(DefElem, cell);
++
++ if (strcmp(defel->defname, "publish_via_parent") == 0)
++ {
++ pubElem = defel;
++ break;
++ }
++ }
++
++ if (pubElem)
++ {
++ bool newSetting = defGetBoolean(pubElem);
++
++ if (newSetting != curSetting)
++ change_publish_via_parent(rel, newSetting);
++ }
++ else if (curSetting)
++ {
++ /* publish_via_parent was RESET */
++ change_publish_via_parent(rel, false);
++ }
++ }
++
+ /*
+ * All we need do here is update the pg_class row; the new options will be
+ * propagated into relcaches during post-commit cache inval.
+@@ src/backend/commands/tablecmds.c: ATExecAddInherit(Relation child_rel, RangeVar *parent, LOCKMODE lockmode)
+ trigger_name, RelationGetRelationName(child_rel)),
+ errdetail("ROW triggers with transition tables are not supported in inheritance hierarchies.")));
+
++ /* publish_via_parent tables are not allowed to add additional parents. */
++ if (RelationIsPublishedViaParent(child_rel))
++ ereport(ERROR,
++ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
++ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
++ "publish_via_parent", RelationGetRelationName(child_rel))));
++
+ /* OK to create inheritance */
+ CreateInheritance(child_rel, parent_rel);
+
+@@ src/backend/commands/tablecmds.c: ATExecDropInherit(Relation rel, RangeVar *parent, LOCKMODE lockmode)
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("cannot change inheritance of a partition")));
+
++ /* publish_via_parent tables are not allowed to remove their parent. */
++ if (RelationIsPublishedViaParent(rel))
++ ereport(ERROR,
++ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
++ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
++ "publish_via_parent", RelationGetRelationName(rel))));
++
+ /*
+ * AccessShareLock on the parent is probably enough, seeing that DROP
+ * TABLE doesn't lock parent tables at all. We need some lock since we'll
+
## src/backend/partitioning/partdesc.c ##
@@ src/backend/partitioning/partdesc.c: RelationBuildPartitionDesc(Relation rel, bool omit_detached)
inhoids = find_inheritance_children_extended(RelationGetRelid(rel),
@@ src/backend/replication/logical/tablesync.c: fetch_remote_table_info(char *nspna
+ /*
+ * See if there are any other tables to be copied besides the original. This
+ * happens when a descendant in the inheritance relationship is marked with
-+ * pg_set_logical_root() and is part of a publication with
++ * publish_via_parent and is part of a publication with
+ * publish_via_partition_root = true.
+ *
+ * FIXME: if two publications have conflicting settings for
@@ src/backend/replication/pgoutput/pgoutput.c: pgoutput_change(LogicalDecodingCont
targetrel = ancestor;
}
+ ## src/bin/pg_dump/t/002_pg_dump.pl ##
+@@ src/bin/pg_dump/t/002_pg_dump.pl: my %tests = (
+ no_table_access_method => 1,
+ only_dump_measurement => 1,
+ },
++ },
++
++ 'CREATE TABLE regress_pg_dump_publish_via_parent' => {
++ create_order => 3,
++ create_sql => '
++ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent (col1 int);
++ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1() INHERITS (dump_test.regress_pg_dump_publish_via_parent) WITH (publish_via_parent);',
++ regexp => qr/^
++ \QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent (\E
++ \n\s+\Qcol1 integer\E
++ \n\);
++ (\n.*)+?
++ \n\QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1 (\E
++ \n\)
++ \n\QINHERITS (dump_test.regress_pg_dump_publish_via_parent)\E
++ \n\QWITH (publish_via_parent='true')\E;/xm,
++ like => {
++ %full_runs, %dump_test_schema_runs, section_pre_data => 1,
++ },
++ unlike => {
++ binary_upgrade => 1,
++ exclude_dump_test_schema => 1,
++ only_dump_measurement => 1,
++ },
+ });
+
+ #########################################
+
+ ## src/bin/psql/tab-complete.c ##
+@@ src/bin/psql/tab-complete.c: static const char *const table_storage_parameters[] = {
+ "fillfactor",
+ "log_autovacuum_min_duration",
+ "parallel_workers",
++ "publish_via_parent",
+ "toast.autovacuum_enabled",
+ "toast.autovacuum_freeze_max_age",
+ "toast.autovacuum_freeze_min_age",
+
## src/include/catalog/pg_inherits.h ##
+@@
+
+ #include "nodes/pg_list.h"
+ #include "storage/lock.h"
++#include "utils/relcache.h"
+
+ /* ----------------
+ * pg_inherits definition. cpp turns this into
@@ src/include/catalog/pg_inherits.h: typedef FormData_pg_inherits *Form_pg_inherits;
DECLARE_UNIQUE_INDEX_PKEY(pg_inherits_relid_seqno_index, 2680, InheritsRelidSeqnoIndexId, on pg_inherits using btree(inhrelid oid_ops, inhseqno int4_ops));
@@ src/include/catalog/pg_proc.dat
{ oid => '8139',
descr => 'get information on how a relation will be published via a list of publications',
proname => 'pg_get_relation_publishing_info', provariadic => 'text',
-@@
- proname => 'pg_partition_root', prorettype => 'regclass',
- proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+
+ ## src/include/utils/rel.h ##
+@@ src/include/utils/rel.h: typedef struct StdRdOptions
+ int toast_tuple_target; /* target for tuple toasting */
+ AutoVacOpts autovacuum; /* autovacuum-related options */
+ bool user_catalog_table; /* use as an additional catalog relation */
++ bool publish_via_parent; /* publish via parent's relid for logrep */
+ int parallel_workers; /* max number of parallel workers */
+ StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */
+ bool vacuum_truncate; /* enables vacuum to truncate a relation */
+@@ src/include/utils/rel.h: typedef struct StdRdOptions
+ (relation)->rd_rel->relkind == RELKIND_MATVIEW) ? \
+ ((StdRdOptions *) (relation)->rd_options)->user_catalog_table : false)
-+{ oid => '8136', descr => 'mark a table root for logical replication',
-+ proname => 'pg_set_logical_root', provolatile => 'v', proparallel => 'u',
-+ prorettype => 'void', proargtypes => 'regclass regclass',
-+ prosrc => 'pg_set_logical_root' },
++/*
++ * RelationIsPublishedViaParent
++ * Returns whether the relation's contents should be logically replicated
++ * via its parent table's relid, when published under the effects of
++ * publish_via_partition_root. Note multiple eval of argument!
++ */
++#define RelationIsPublishedViaParent(relation) \
++ ((relation)->rd_options && \
++ ((relation)->rd_rel->relkind == RELKIND_RELATION) && \
++ ((StdRdOptions *) (relation)->rd_options)->publish_via_parent)
+
- { oid => '4350', descr => 'Unicode normalization',
- proname => 'normalize', prorettype => 'text', proargtypes => 'text text',
- prosrc => 'unicode_normalize_func' },
+ /*
+ * RelationGetParallelWorkers
+ * Returns the relation's parallel_workers reloption setting.
## src/test/regress/expected/publication.out ##
@@ src/test/regress/expected/publication.out: CREATE ROLE regress_publication_user LOGIN SUPERUSER;
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
-+-- Sanity check cases for pg_set_logical_root().
++-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
++-- should do nothing at all
++ALTER TABLE sch1.iroot SET (publish_via_parent = false);
++ALTER TABLE sch1.iroot RESET (publish_via_parent);
+-- marking roots between unrelated tables is not allowed
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
-+ERROR: table "ipart1" does not inherit from intended root table "iroot"
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
-+ERROR: table "ipart2" does not inherit from intended root table "tbl1"
++ALTER TABLE sch1.ipart1 SET (publish_via_parent);
++ERROR: table "ipart1" does not inherit from any tables
++CREATE TABLE fail (a int) WITH (publish_via_parent);
++ERROR: table "fail" does not inherit from any tables
+-- establishing an inheritance relationship fixes the problem
-+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
++ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
++ SET (publish_via_parent);
+-- but multiple inheritance is not allowed
-+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
++ SET (publish_via_parent);
+ERROR: table "ipart2" inherits from multiple tables
-+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
++-- once publish_via_parent is set, inheritance cannot be changed
++ALTER TABLE sch1.ipart2 SET (publish_via_parent);
++ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
++ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
++ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should fail
++ALTER TABLE part SET (publish_via_parent); -- should fail
+ERROR: must be owner of table part
+RESET SESSION AUTHORIZATION;
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should also fail
++ALTER TABLE part SET (publish_via_parent); -- should also fail
++ERROR: must be owner of table root
++CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+ERROR: must be owner of table root
+RESET SESSION AUTHORIZATION;
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
-+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
-+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1);
++CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
-+SELECT pg_set_logical_root('sch1.ipart1_a', 'sch1.ipart1');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
-+SELECT pg_set_logical_root('sch1.ipart1_a1', 'sch1.ipart1_a');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
++ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
+ WITH (publish_via_partition_root);
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ ipub_other | sch1.iroot | sch1.ipart1_a
+(10 rows)
+
++-- "Detaching" partitions should change the subscriptions
++ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++ pubname | relid
++------------+-------------
++ ipub_root | sch1.iroot
++ ipub_root | sch1.ipart1
++ ipub_root | sch1.ipart2
++ ipub_part1 | sch1.ipart1
++ ipub_other | sch1.iroot
++ ipub_other | sch1.ipart2
++(6 rows)
++
++SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
++ published | synced
++-------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.iroot | sch1.ipart1_a
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1 | sch1.ipart1_a1
++ sch1.ipart2 | sch1.ipart2
++(5 rows)
++
++SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++ pubname | published | synced
++------------+-------------+----------------
++ ipub_root | sch1.iroot | sch1.iroot
++ ipub_root | sch1.iroot | sch1.ipart1_a
++ ipub_root | sch1.ipart1 | sch1.ipart1
++ ipub_root | sch1.ipart1 | sch1.ipart1_a1
++ ipub_root | sch1.ipart2 | sch1.ipart2
++ ipub_part1 | sch1.ipart1 | sch1.ipart1
++ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
++ ipub_other | sch1.iroot | sch1.iroot
++ ipub_other | sch1.iroot | sch1.ipart1_a
++ ipub_other | sch1.ipart2 | sch1.ipart2
++(10 rows)
++
++ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
++ relid
++----------------
++ sch1.ipart1
++ sch1.ipart1_a1
++(2 rows)
++
++SELECT * FROM published_sync('ipub_part1');
++ published | synced
++----------------+----------------
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1_a1 | sch1.ipart1_a1
++(2 rows)
++
++SELECT * FROM published_stream('ipub_part1');
++ pubname | published | synced
++------------+----------------+----------------
++ ipub_part1 | sch1.ipart1 | sch1.ipart1
++ ipub_part1 | sch1.ipart1_a1 | sch1.ipart1_a1
++(2 rows)
++
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
@@ src/test/regress/sql/publication.sql: ALTER TABLE sch1.tbl1 ATTACH PARTITION sch
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
-+-- Sanity check cases for pg_set_logical_root().
++-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+
++-- should do nothing at all
++ALTER TABLE sch1.iroot SET (publish_via_parent = false);
++ALTER TABLE sch1.iroot RESET (publish_via_parent);
++
+-- marking roots between unrelated tables is not allowed
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
++ALTER TABLE sch1.ipart1 SET (publish_via_parent);
++CREATE TABLE fail (a int) WITH (publish_via_parent);
+
+-- establishing an inheritance relationship fixes the problem
-+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
++ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
++ SET (publish_via_parent);
+
+-- but multiple inheritance is not allowed
-+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
++ SET (publish_via_parent);
+
-+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
++-- once publish_via_parent is set, inheritance cannot be changed
++ALTER TABLE sch1.ipart2 SET (publish_via_parent);
++ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
++
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should fail
++ALTER TABLE part SET (publish_via_parent); -- should fail
+RESET SESSION AUTHORIZATION;
++
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should also fail
++ALTER TABLE part SET (publish_via_parent); -- should also fail
++CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+RESET SESSION AUTHORIZATION;
++
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+
-+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
-+
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
-+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1);
++CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
-+
-+SELECT pg_set_logical_root('sch1.ipart1_a', 'sch1.ipart1');
-+SELECT pg_set_logical_root('sch1.ipart1_a1', 'sch1.ipart1_a');
++ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
@@ src/test/regress/sql/publication.sql: ALTER TABLE sch1.tbl1 ATTACH PARTITION sch
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+
++-- "Detaching" partitions should change the subscriptions
++ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
++SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++
++ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
++SELECT * FROM published_sync('ipub_part1');
++SELECT * FROM published_stream('ipub_part1');
++
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
@@ src/test/subscription/t/013_partition.pl: $node_subscriber1->append_conf('postgr
$node_subscriber1->reload;
+# Make sure standard inheritance setups aren't broken by the new
-+# pg_set_logical_root() handling.
++# publish_via_parent handling.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
@@ src/test/subscription/t/013_partition.pl: $node_subscriber1->append_conf('postgr
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
++ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
++ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_1 VALUES (1, 'itab1')");
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+ FOR EACH ROW EXECUTE FUNCTION itab1_trigger();");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
++ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
++ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+ FOR EACH ROW EXECUTE FUNCTION itab2_trigger();");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab2_1', 'itab2')");
++ "ALTER TABLE itab2_1 SET (publish_via_parent = true)");
+
+# itab2_1 should be published using its own identity here, since its parent is
+# not included. itab1_1 should be published via its parent, itab1, without
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab3_1', 'itab3')");
++ "ALTER TABLE itab3_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab3_1_1', 'itab3_1')");
++ "ALTER TABLE itab3_1_1 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3 VALUES (1, 'itab3')");
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+is($result, qq(), 'initial data routed for itab3_1_1 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
-+$node_publisher->safe_psql('postgres', "INSERT INTO itab3 VALUES (4, 'itab3-new')");
-+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1 VALUES (4, 'itab3_1-new')");
-+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1_1 VALUES (4, 'itab3_1_1-new')");
++$node_publisher->safe_psql('postgres', "INSERT INTO itab3 VALUES (4, 'itab3_new')");
++$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1 VALUES (4, 'itab3_1_new')");
++$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1_1 VALUES (4, 'itab3_1_1_new')");
+
+$node_publisher->wait_for_catchup('mixed');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3 ORDER BY 1, 2");
+is($result, qq(1|itab3
-+4|itab3-new), 'new data routed for itab3 on subscriber 2');
++4|itab3_new), 'new data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1
-+4|itab3_1-new
-+4|itab3_1_1-new), 'new data routed for itab3_1 on subscriber 2');
++4|itab3_1_1_new
++4|itab3_1_new), 'new data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
Attachments:
[text/plain] since-v2.diff.txt (41.4K, ../../[email protected]/2-since-v2.diff.txt)
download | inline:
1: 761b91b3f1 = 1: 7e68f96af6 pgoutput: refactor publication cache construction
2: 0397a52583 ! 2: a48563919e WIP: introduce pg_set_logical_root for use with pubviaroot
@@ Metadata
Author: Jacob Champion <[email protected]>
## Commit message ##
- WIP: introduce pg_set_logical_root for use with pubviaroot
+ WIP: introduce publish_via_parent for logical replication
- Allows regular inherited tables to be published via their root table,
- just like partitions. This works by hijacking pg_inherit's inhseqno
- column, and replacing a (single) existing entry for the child with the
- value zero, indicating that it should be treated as a logical partition
- by the publication machinery.
+ This new relation option allows regular inherited tables to be published
+ via their root table, just like partitions. This works by hijacking
+ pg_inherit's inhseqno column, and replacing a (single) existing entry
+ for the child with the value zero, indicating that it should be treated
+ as a logical partition by the publication machinery.
Initial sync works by asking the publisher for a list of logical
descendants of the published table, then COPYing them one-by-one into
@@ Commit message
roots.
Known bugs/TODOs:
- - The pg_inherits machinery doesn't prohibit changes to inheritance
- after an entry has been marked as a logical root.
+ - If two publications publish the same leaves of a table hierarchy, but
+ have different roots, the shared leaf tables will be incorrectly
+ duplicated when subscribing to both publications simultaneously.
- I haven't given any thought to interactions with row filters, or to
column lists.
- I'm not sure that I'm taking all the necessary locks yet, and those I
do take may be taken in the wrong order.
- - Dump and upgrade aren't supported yet.
+
+ ## src/backend/access/common/reloptions.c ##
+@@ src/backend/access/common/reloptions.c: static relopt_bool boolRelOpts[] =
+ },
+ false
+ },
++ {
++ {
++ "publish_via_parent",
++ "Replicate this table's contents via its parent table when publishing with publish_via_partition_root",
++ RELOPT_KIND_HEAP,
++ AccessExclusiveLock
++ },
++ false
++ },
+ {
+ {
+ "fastupdate",
+@@ src/backend/access/common/reloptions.c: default_reloptions(Datum reloptions, bool validate, relopt_kind kind)
+ offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, analyze_scale_factor)},
+ {"user_catalog_table", RELOPT_TYPE_BOOL,
+ offsetof(StdRdOptions, user_catalog_table)},
++ {"publish_via_parent", RELOPT_TYPE_BOOL,
++ offsetof(StdRdOptions, publish_via_parent)},
+ {"parallel_workers", RELOPT_TYPE_INT,
+ offsetof(StdRdOptions, parallel_workers)},
+ {"vacuum_index_cleanup", RELOPT_TYPE_ENUM,
## src/backend/catalog/pg_inherits.c ##
@@
@@ src/backend/catalog/pg_inherits.c: find_inheritance_children(Oid parentrelId, LO
* committed to the active snapshot. In addition, *detached_xmin (if not null)
* is set to the xmin of the row of the detached partition.
+ *
-+ * If logical_only is true, only tables marked explicitly via
-+ * pg_set_logical_root() are included in the output list.
++ * If logical_only is true, only tables marked explicitly via publish_via_parent
++ * are included in the output list.
*/
List *
find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
@@ src/backend/catalog/pg_inherits.c: PartitionHasPendingDetach(Oid partoid)
+has_logical_parent(Relation inhRel, Oid relid)
+{
+ return (get_logical_parent_worker(inhRel, relid) != InvalidOid);
-+}
-+
-+Datum
-+pg_set_logical_root(PG_FUNCTION_ARGS)
-+{
-+ Oid tableoid = PG_GETARG_OID(0);
-+ Oid rootoid = PG_GETARG_OID(1);
-+ char *tablename;
-+ char *rootname;
-+ Relation inhRel;
-+ ScanKeyData key;
-+ SysScanDesc scan;
-+ Oid parent = InvalidOid;
-+ HeapTuple tuple, copyTuple;
-+ Form_pg_inherits form;
-+
-+ /*
-+ * Check that the tables exist.
-+ * TODO: check identical schemas too? or does replication handle that?
-+ */
-+ tablename = get_rel_name(tableoid);
-+ if (tablename == NULL)
-+ ereport(ERROR,
-+ (errcode(ERRCODE_UNDEFINED_TABLE),
-+ errmsg("OID %u does not refer to a table", tableoid)));
-+ rootname = get_rel_name(rootoid);
-+ if (rootname == NULL)
-+ ereport(ERROR,
-+ (errcode(ERRCODE_UNDEFINED_TABLE),
-+ errmsg("OID %u does not refer to a table", rootoid)));
-+
-+ /* Check ownership. */
-+ if (!object_ownercheck(RelationRelationId, tableoid, GetUserId()))
-+ aclcheck_error(ACLCHECK_NOT_OWNER,
-+ get_relkind_objtype(get_rel_relkind(tableoid)),
-+ tablename);
-+ if (!object_ownercheck(RelationRelationId, rootoid, GetUserId()))
-+ aclcheck_error(ACLCHECK_NOT_OWNER,
-+ get_relkind_objtype(get_rel_relkind(rootoid)),
-+ rootname);
-+
-+ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
-+ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
-+
-+ /*
-+ * We have to make sure that the inheritance relationship already exists,
-+ * and that there is only one existing parent for this table.
-+ *
-+ * TODO: do we have to lock the tables themselves to avoid races?
-+ */
-+ ScanKeyInit(&key,
-+ Anum_pg_inherits_inhrelid,
-+ BTEqualStrategyNumber, F_OIDEQ,
-+ ObjectIdGetDatum(tableoid));
-+
-+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
-+ NULL, 1, &key);
-+ tuple = systable_getnext(scan);
-+ if (HeapTupleIsValid(tuple))
-+ {
-+ form = (Form_pg_inherits) GETSTRUCT(tuple);
-+ parent = form->inhparent;
-+ copyTuple = heap_copytuple(tuple);
-+
-+ if (HeapTupleIsValid(systable_getnext(scan)))
-+ ereport(ERROR,
-+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
-+ errmsg("table \"%s\" inherits from multiple tables",
-+ tablename)));
-+ }
-+
-+ if (parent != rootoid)
-+ ereport(ERROR,
-+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
-+ errmsg("table \"%s\" does not inherit from intended root table \"%s\"",
-+ tablename, rootname)));
-+
-+ systable_endscan(scan);
-+
-+ /* Mark the inheritance as a logical root by setting it to zero. */
-+ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
-+ form->inhseqno = 0;
-+
-+ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
-+
-+ heap_freetuple(copyTuple);
-+ table_close(inhRel, RowExclusiveLock);
-+
-+ PG_RETURN_VOID();
+}
## src/backend/catalog/pg_publication.c ##
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
+ * The only time this list consists of anything more than the table itself is
+ * when a publication's publish_via_partition_root is set to true and the table
+ * has inherited child tables in the publications that have been marked with
-+ * pg_set_logical_root().
++ * publish_via_parent.
+ */
+Datum
+pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
+ * Check if any of the logical ancestors (that is, partition
-+ * parents or tables marked with pg_set_logical_root()) are
++ * parents or tables marked with publish_via_parent) are
+ * published. If so, note down the topmost ancestor that is
* published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
return rel_publications;
}
+@@ src/backend/catalog/pg_publication.c: pg_get_relation_publishing_info(PG_FUNCTION_ARGS)
+
+ {
+ List *rel_publications;
+- PublicationActions pubactions;
++ PublicationActions pubactions = {0};
+ Oid publish_as_relid;
+ ListCell *lc;
+ Datum *puboids;
## src/backend/commands/publicationcmds.c ##
@@ src/backend/commands/publicationcmds.c: contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
@@ src/backend/commands/publicationcmds.c: contain_invalid_rfcolumn_walker(Node *no
* information of the child table. So, get the column number of the
* child table as parent and child column order could be different.
+ *
-+ * TODO: is this applicable to pg_set_logical_root()?
++ * TODO: is this applicable to publish_via_parent?
*/
if (context->pubviaroot)
{
@@ src/backend/commands/publicationcmds.c: pub_rf_contains_invalid_column(Oid pubid
* Note that even though the row filter used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
+ *
-+ * TODO: is this applicable to pg_set_logical_root()?
++ * TODO: is this applicable to publish_via_parent?
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
@@ src/backend/commands/publicationcmds.c: pub_rf_contains_invalid_column(Oid pubid
*
* Returns true if any replica identity column is not covered by column list.
+ *
-+ * TODO: pg_set_logical_root()?
++ * TODO: publish_via_parent?
*/
bool
pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
@@ src/backend/commands/publicationcmds.c: TransformPubWhereClauses(List *tables, c
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
-+ * TODO: decide how this interacts with pg_set_logical_root
++ * TODO: decide how this interacts with publish_via_parent
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ src/backend/commands/publicationcmds.c: CheckPubRelationColumnList(char *pubname
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
-+ * TODO: decide if this interacts with pg_set_logical_root()
++ * TODO: decide if this interacts with publish_via_parent
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ## src/backend/commands/tablecmds.c ##
+@@ src/backend/commands/tablecmds.c: static bool MergeCheckConstraint(List *constraints, char *name, Node *expr);
+ static void MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel);
+ static void MergeConstraintsIntoExisting(Relation child_rel, Relation parent_rel);
+ static void StoreCatalogInheritance(Oid relationId, List *supers,
+- bool child_is_partition);
++ bool child_is_partition,
++ bool publish_via_parent);
+ static void StoreCatalogInheritance1(Oid relationId, Oid parentOid,
+ int32 seqNumber, Relation inhRelation,
+ bool child_is_partition);
+@@ src/backend/commands/tablecmds.c: static void ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel,
+ const char *tablespacename, LOCKMODE lockmode);
+ static void ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode);
+ static void ATExecSetTableSpaceNoStorage(Relation rel, Oid newTableSpace);
++static void change_publish_via_parent(Relation rel, bool set);
+ static void ATExecSetRelOptions(Relation rel, List *defList,
+ AlterTableType operation,
+ LOCKMODE lockmode);
+@@ src/backend/commands/tablecmds.c: DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
+ }
+
+ /* Store inheritance information for new rel. */
+- StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL);
++ StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL,
++ RelationIsPublishedViaParent(rel));
+
+ /*
+ * Process the partitioning specification (if any) and store the partition
+@@ src/backend/commands/tablecmds.c: MergeCheckConstraint(List *constraints, char *name, Node *expr)
+ */
+ static void
+ StoreCatalogInheritance(Oid relationId, List *supers,
+- bool child_is_partition)
++ bool child_is_partition, bool publish_via_parent)
+ {
+ Relation relation;
+ int32 seqNumber;
+@@ src/backend/commands/tablecmds.c: StoreCatalogInheritance(Oid relationId, List *supers,
+ */
+ Assert(OidIsValid(relationId));
+
++ /*
++ * publish_via_parent requires exactly one parent. Try to keep this
++ * messaging consistent with ALTER TABLE/change_publish_via_parent().
++ *
++ * Earlier, in MergeAttributes, we already checked that we had ownership of
++ * the parent table, so we don't need to check that again the way that
++ * change_publish_via_parent() does.
++ */
++ if (publish_via_parent && list_length(supers) != 1)
++ ereport(ERROR,
++ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
++ errmsg((list_length(supers) == 0)
++ ? "table \"%s\" does not inherit from any tables"
++ : "table \"%s\" inherits from multiple tables",
++ get_rel_name(relationId))));
++
+ if (supers == NIL)
+ return;
+
+@@ src/backend/commands/tablecmds.c: StoreCatalogInheritance(Oid relationId, List *supers,
+ */
+ relation = table_open(InheritsRelationId, RowExclusiveLock);
+
+- seqNumber = 1;
++ /*
++ * Normally inhseqno starts at one, but for publish_via_parent, it's given a
++ * sentinel value of zero. (In that case, we'll only go through this loop
++ * once, as the list_length() check above enforces.)
++ */
++ seqNumber = publish_via_parent ? 0 : 1;
+ foreach(entry, supers)
+ {
+ Oid parentOid = lfirst_oid(entry);
+@@ src/backend/commands/tablecmds.c: ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel, const char *tablespacen
+ tab->newTableSpace = tablespaceId;
+ }
+
++static void
++change_publish_via_parent(Relation rel, bool set)
++{
++ Relation inhRel;
++ ScanKeyData key;
++ SysScanDesc scan;
++ Oid parentOid;
++ Relation parentRel;
++ HeapTuple tuple, copyTuple;
++ Form_pg_inherits form;
++
++ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
++ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
++
++ /*
++ * We have to make sure that an inheritance relationship already exists, and
++ * that there is only one candidate parent for this table.
++ *
++ * TODO: do we have to lock the tables themselves to avoid races?
++ */
++ ScanKeyInit(&key,
++ Anum_pg_inherits_inhrelid,
++ BTEqualStrategyNumber, F_OIDEQ,
++ ObjectIdGetDatum(RelationGetRelid(rel)));
++
++ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
++ NULL, 1, &key);
++ tuple = systable_getnext(scan);
++
++ if (!HeapTupleIsValid(tuple))
++ ereport(ERROR,
++ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
++ errmsg("table \"%s\" does not inherit from any tables",
++ RelationGetRelationName(rel))));
++
++ form = (Form_pg_inherits) GETSTRUCT(tuple);
++ parentOid = form->inhparent;
++ copyTuple = heap_copytuple(tuple);
++
++ if (HeapTupleIsValid(systable_getnext(scan)))
++ ereport(ERROR,
++ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
++ errmsg("table \"%s\" inherits from multiple tables",
++ RelationGetRelationName(rel))));
++
++ systable_endscan(scan);
++
++ if (set)
++ {
++ /*
++ * Open our parent table and check permissions. Like ATTACH PARTITION,
++ * we require ownership of the parent table in addition to the child
++ * (which has already been checked in ATPrepCmd).
++ *
++ * TODO: what lock strength is needed here? Is nesting it inside the
++ * inhRel lock a risk?
++ */
++ parentRel = table_open(parentOid, AccessExclusiveLock);
++ ATSimplePermissions(AT_SetRelOptions, parentRel, ATT_TABLE);
++ table_close(parentRel, NoLock); /* TODO: do we need to hang onto the lock? */
++ }
++ else
++ {
++ /*
++ * Ownership of the parent isn't needed for unsetting the flag, just
++ * like we wouldn't check ownership during NO INHERIT.
++ */
++ }
++
++ /*
++ * Mark the inheritance as a logical root by setting inhseqno to zero.
++ * Unmark it by setting inhseqno to one (since there's only one parent).
++ */
++ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
++ form->inhseqno = set ? 0 : 1;
++
++ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
++
++ heap_freetuple(copyTuple);
++ table_close(inhRel, RowExclusiveLock);
++}
++
+ /*
+ * Set, reset, or replace reloptions.
+ */
+@@ src/backend/commands/tablecmds.c: ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation,
+ }
+ }
+
++ /* Extra work for publish_via_parent */
++ if (rel->rd_rel->relkind == RELKIND_RELATION)
++ {
++ List *heap_options = untransformRelOptions(newOptions);
++ ListCell *cell;
++ bool curSetting = RelationIsPublishedViaParent(rel);
++ DefElem *pubElem = NULL;
++
++ foreach (cell, heap_options)
++ {
++ DefElem *defel = lfirst_node(DefElem, cell);
++
++ if (strcmp(defel->defname, "publish_via_parent") == 0)
++ {
++ pubElem = defel;
++ break;
++ }
++ }
++
++ if (pubElem)
++ {
++ bool newSetting = defGetBoolean(pubElem);
++
++ if (newSetting != curSetting)
++ change_publish_via_parent(rel, newSetting);
++ }
++ else if (curSetting)
++ {
++ /* publish_via_parent was RESET */
++ change_publish_via_parent(rel, false);
++ }
++ }
++
+ /*
+ * All we need do here is update the pg_class row; the new options will be
+ * propagated into relcaches during post-commit cache inval.
+@@ src/backend/commands/tablecmds.c: ATExecAddInherit(Relation child_rel, RangeVar *parent, LOCKMODE lockmode)
+ trigger_name, RelationGetRelationName(child_rel)),
+ errdetail("ROW triggers with transition tables are not supported in inheritance hierarchies.")));
+
++ /* publish_via_parent tables are not allowed to add additional parents. */
++ if (RelationIsPublishedViaParent(child_rel))
++ ereport(ERROR,
++ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
++ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
++ "publish_via_parent", RelationGetRelationName(child_rel))));
++
+ /* OK to create inheritance */
+ CreateInheritance(child_rel, parent_rel);
+
+@@ src/backend/commands/tablecmds.c: ATExecDropInherit(Relation rel, RangeVar *parent, LOCKMODE lockmode)
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("cannot change inheritance of a partition")));
+
++ /* publish_via_parent tables are not allowed to remove their parent. */
++ if (RelationIsPublishedViaParent(rel))
++ ereport(ERROR,
++ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
++ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
++ "publish_via_parent", RelationGetRelationName(rel))));
++
+ /*
+ * AccessShareLock on the parent is probably enough, seeing that DROP
+ * TABLE doesn't lock parent tables at all. We need some lock since we'll
+
## src/backend/partitioning/partdesc.c ##
@@ src/backend/partitioning/partdesc.c: RelationBuildPartitionDesc(Relation rel, bool omit_detached)
inhoids = find_inheritance_children_extended(RelationGetRelid(rel),
@@ src/backend/replication/logical/tablesync.c: fetch_remote_table_info(char *nspna
+ /*
+ * See if there are any other tables to be copied besides the original. This
+ * happens when a descendant in the inheritance relationship is marked with
-+ * pg_set_logical_root() and is part of a publication with
++ * publish_via_parent and is part of a publication with
+ * publish_via_partition_root = true.
+ *
+ * FIXME: if two publications have conflicting settings for
@@ src/backend/replication/pgoutput/pgoutput.c: pgoutput_change(LogicalDecodingCont
targetrel = ancestor;
}
+ ## src/bin/pg_dump/t/002_pg_dump.pl ##
+@@ src/bin/pg_dump/t/002_pg_dump.pl: my %tests = (
+ no_table_access_method => 1,
+ only_dump_measurement => 1,
+ },
++ },
++
++ 'CREATE TABLE regress_pg_dump_publish_via_parent' => {
++ create_order => 3,
++ create_sql => '
++ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent (col1 int);
++ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1() INHERITS (dump_test.regress_pg_dump_publish_via_parent) WITH (publish_via_parent);',
++ regexp => qr/^
++ \QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent (\E
++ \n\s+\Qcol1 integer\E
++ \n\);
++ (\n.*)+?
++ \n\QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1 (\E
++ \n\)
++ \n\QINHERITS (dump_test.regress_pg_dump_publish_via_parent)\E
++ \n\QWITH (publish_via_parent='true')\E;/xm,
++ like => {
++ %full_runs, %dump_test_schema_runs, section_pre_data => 1,
++ },
++ unlike => {
++ binary_upgrade => 1,
++ exclude_dump_test_schema => 1,
++ only_dump_measurement => 1,
++ },
+ });
+
+ #########################################
+
+ ## src/bin/psql/tab-complete.c ##
+@@ src/bin/psql/tab-complete.c: static const char *const table_storage_parameters[] = {
+ "fillfactor",
+ "log_autovacuum_min_duration",
+ "parallel_workers",
++ "publish_via_parent",
+ "toast.autovacuum_enabled",
+ "toast.autovacuum_freeze_max_age",
+ "toast.autovacuum_freeze_min_age",
+
## src/include/catalog/pg_inherits.h ##
+@@
+
+ #include "nodes/pg_list.h"
+ #include "storage/lock.h"
++#include "utils/relcache.h"
+
+ /* ----------------
+ * pg_inherits definition. cpp turns this into
@@ src/include/catalog/pg_inherits.h: typedef FormData_pg_inherits *Form_pg_inherits;
DECLARE_UNIQUE_INDEX_PKEY(pg_inherits_relid_seqno_index, 2680, InheritsRelidSeqnoIndexId, on pg_inherits using btree(inhrelid oid_ops, inhseqno int4_ops));
@@ src/include/catalog/pg_proc.dat
{ oid => '8139',
descr => 'get information on how a relation will be published via a list of publications',
proname => 'pg_get_relation_publishing_info', provariadic => 'text',
-@@
- proname => 'pg_partition_root', prorettype => 'regclass',
- proargtypes => 'regclass', prosrc => 'pg_partition_root' },
+
+ ## src/include/utils/rel.h ##
+@@ src/include/utils/rel.h: typedef struct StdRdOptions
+ int toast_tuple_target; /* target for tuple toasting */
+ AutoVacOpts autovacuum; /* autovacuum-related options */
+ bool user_catalog_table; /* use as an additional catalog relation */
++ bool publish_via_parent; /* publish via parent's relid for logrep */
+ int parallel_workers; /* max number of parallel workers */
+ StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */
+ bool vacuum_truncate; /* enables vacuum to truncate a relation */
+@@ src/include/utils/rel.h: typedef struct StdRdOptions
+ (relation)->rd_rel->relkind == RELKIND_MATVIEW) ? \
+ ((StdRdOptions *) (relation)->rd_options)->user_catalog_table : false)
-+{ oid => '8136', descr => 'mark a table root for logical replication',
-+ proname => 'pg_set_logical_root', provolatile => 'v', proparallel => 'u',
-+ prorettype => 'void', proargtypes => 'regclass regclass',
-+ prosrc => 'pg_set_logical_root' },
++/*
++ * RelationIsPublishedViaParent
++ * Returns whether the relation's contents should be logically replicated
++ * via its parent table's relid, when published under the effects of
++ * publish_via_partition_root. Note multiple eval of argument!
++ */
++#define RelationIsPublishedViaParent(relation) \
++ ((relation)->rd_options && \
++ ((relation)->rd_rel->relkind == RELKIND_RELATION) && \
++ ((StdRdOptions *) (relation)->rd_options)->publish_via_parent)
+
- { oid => '4350', descr => 'Unicode normalization',
- proname => 'normalize', prorettype => 'text', proargtypes => 'text text',
- prosrc => 'unicode_normalize_func' },
+ /*
+ * RelationGetParallelWorkers
+ * Returns the relation's parallel_workers reloption setting.
## src/test/regress/expected/publication.out ##
@@ src/test/regress/expected/publication.out: CREATE ROLE regress_publication_user LOGIN SUPERUSER;
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
-+-- Sanity check cases for pg_set_logical_root().
++-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
++-- should do nothing at all
++ALTER TABLE sch1.iroot SET (publish_via_parent = false);
++ALTER TABLE sch1.iroot RESET (publish_via_parent);
+-- marking roots between unrelated tables is not allowed
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
-+ERROR: table "ipart1" does not inherit from intended root table "iroot"
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
-+ERROR: table "ipart2" does not inherit from intended root table "tbl1"
++ALTER TABLE sch1.ipart1 SET (publish_via_parent);
++ERROR: table "ipart1" does not inherit from any tables
++CREATE TABLE fail (a int) WITH (publish_via_parent);
++ERROR: table "fail" does not inherit from any tables
+-- establishing an inheritance relationship fixes the problem
-+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
++ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
++ SET (publish_via_parent);
+-- but multiple inheritance is not allowed
-+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
++ SET (publish_via_parent);
+ERROR: table "ipart2" inherits from multiple tables
-+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
++-- once publish_via_parent is set, inheritance cannot be changed
++ALTER TABLE sch1.ipart2 SET (publish_via_parent);
++ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
++ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
++ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should fail
++ALTER TABLE part SET (publish_via_parent); -- should fail
+ERROR: must be owner of table part
+RESET SESSION AUTHORIZATION;
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should also fail
++ALTER TABLE part SET (publish_via_parent); -- should also fail
++ERROR: must be owner of table root
++CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+ERROR: must be owner of table root
+RESET SESSION AUTHORIZATION;
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
-+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
-+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1);
++CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
-+SELECT pg_set_logical_root('sch1.ipart1_a', 'sch1.ipart1');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
-+SELECT pg_set_logical_root('sch1.ipart1_a1', 'sch1.ipart1_a');
-+ pg_set_logical_root
-+---------------------
-+
-+(1 row)
-+
++ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
+ WITH (publish_via_partition_root);
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ ipub_other | sch1.iroot | sch1.ipart1_a
+(10 rows)
+
++-- "Detaching" partitions should change the subscriptions
++ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++ pubname | relid
++------------+-------------
++ ipub_root | sch1.iroot
++ ipub_root | sch1.ipart1
++ ipub_root | sch1.ipart2
++ ipub_part1 | sch1.ipart1
++ ipub_other | sch1.iroot
++ ipub_other | sch1.ipart2
++(6 rows)
++
++SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
++ published | synced
++-------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.iroot | sch1.ipart1_a
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1 | sch1.ipart1_a1
++ sch1.ipart2 | sch1.ipart2
++(5 rows)
++
++SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++ pubname | published | synced
++------------+-------------+----------------
++ ipub_root | sch1.iroot | sch1.iroot
++ ipub_root | sch1.iroot | sch1.ipart1_a
++ ipub_root | sch1.ipart1 | sch1.ipart1
++ ipub_root | sch1.ipart1 | sch1.ipart1_a1
++ ipub_root | sch1.ipart2 | sch1.ipart2
++ ipub_part1 | sch1.ipart1 | sch1.ipart1
++ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
++ ipub_other | sch1.iroot | sch1.iroot
++ ipub_other | sch1.iroot | sch1.ipart1_a
++ ipub_other | sch1.ipart2 | sch1.ipart2
++(10 rows)
++
++ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
++ relid
++----------------
++ sch1.ipart1
++ sch1.ipart1_a1
++(2 rows)
++
++SELECT * FROM published_sync('ipub_part1');
++ published | synced
++----------------+----------------
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1_a1 | sch1.ipart1_a1
++(2 rows)
++
++SELECT * FROM published_stream('ipub_part1');
++ pubname | published | synced
++------------+----------------+----------------
++ ipub_part1 | sch1.ipart1 | sch1.ipart1
++ ipub_part1 | sch1.ipart1_a1 | sch1.ipart1_a1
++(2 rows)
++
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
@@ src/test/regress/sql/publication.sql: ALTER TABLE sch1.tbl1 ATTACH PARTITION sch
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
-+-- Sanity check cases for pg_set_logical_root().
++-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+
++-- should do nothing at all
++ALTER TABLE sch1.iroot SET (publish_via_parent = false);
++ALTER TABLE sch1.iroot RESET (publish_via_parent);
++
+-- marking roots between unrelated tables is not allowed
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.tbl1');
++ALTER TABLE sch1.ipart1 SET (publish_via_parent);
++CREATE TABLE fail (a int) WITH (publish_via_parent);
+
+-- establishing an inheritance relationship fixes the problem
-+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot;
-+SELECT pg_set_logical_root('sch1.ipart1', 'sch1.iroot');
++ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
++ SET (publish_via_parent);
+
+-- but multiple inheritance is not allowed
-+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
++ SET (publish_via_parent);
+
-+ALTER TABLE sch1.ipart2 NO INHERIT sch1.ipart1;
-+SELECT pg_set_logical_root('sch1.ipart2', 'sch1.iroot');
++-- once publish_via_parent is set, inheritance cannot be changed
++ALTER TABLE sch1.ipart2 SET (publish_via_parent);
++ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
++ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
++
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should fail
++ALTER TABLE part SET (publish_via_parent); -- should fail
+RESET SESSION AUTHORIZATION;
++
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
-+SELECT pg_set_logical_root('part', 'root'); -- should also fail
++ALTER TABLE part SET (publish_via_parent); -- should also fail
++CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+RESET SESSION AUTHORIZATION;
++
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+
-+-- TODO: make sure existing logical descendant can't be ALTERed [NO] INHERIT
-+
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
-+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1);
++CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
-+
-+SELECT pg_set_logical_root('sch1.ipart1_a', 'sch1.ipart1');
-+SELECT pg_set_logical_root('sch1.ipart1_a1', 'sch1.ipart1_a');
++ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
@@ src/test/regress/sql/publication.sql: ALTER TABLE sch1.tbl1 ATTACH PARTITION sch
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+
++-- "Detaching" partitions should change the subscriptions
++ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
++SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++
++ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
++SELECT * FROM published_sync('ipub_part1');
++SELECT * FROM published_stream('ipub_part1');
++
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
@@ src/test/subscription/t/013_partition.pl: $node_subscriber1->append_conf('postgr
$node_subscriber1->reload;
+# Make sure standard inheritance setups aren't broken by the new
-+# pg_set_logical_root() handling.
++# publish_via_parent handling.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
@@ src/test/subscription/t/013_partition.pl: $node_subscriber1->append_conf('postgr
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
++ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
++ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_1 VALUES (1, 'itab1')");
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+ FOR EACH ROW EXECUTE FUNCTION itab1_trigger();");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_1', 'itab1')");
++ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab1_2', 'itab1')");
++ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+ FOR EACH ROW EXECUTE FUNCTION itab2_trigger();");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab2_1', 'itab2')");
++ "ALTER TABLE itab2_1 SET (publish_via_parent = true)");
+
+# itab2_1 should be published using its own identity here, since its parent is
+# not included. itab1_1 should be published via its parent, itab1, without
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab3_1', 'itab3')");
++ "ALTER TABLE itab3_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
-+ "SELECT pg_set_logical_root('itab3_1_1', 'itab3_1')");
++ "ALTER TABLE itab3_1_1 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3 VALUES (1, 'itab3')");
@@ src/test/subscription/t/013_partition.pl: $result = $node_subscriber2->safe_psql
+is($result, qq(), 'initial data routed for itab3_1_1 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
-+$node_publisher->safe_psql('postgres', "INSERT INTO itab3 VALUES (4, 'itab3-new')");
-+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1 VALUES (4, 'itab3_1-new')");
-+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1_1 VALUES (4, 'itab3_1_1-new')");
++$node_publisher->safe_psql('postgres', "INSERT INTO itab3 VALUES (4, 'itab3_new')");
++$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1 VALUES (4, 'itab3_1_new')");
++$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1_1 VALUES (4, 'itab3_1_1_new')");
+
+$node_publisher->wait_for_catchup('mixed');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3 ORDER BY 1, 2");
+is($result, qq(1|itab3
-+4|itab3-new), 'new data routed for itab3 on subscriber 2');
++4|itab3_new), 'new data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1
-+4|itab3_1-new
-+4|itab3_1_1-new), 'new data routed for itab3_1 on subscriber 2');
++4|itab3_1_1_new
++4|itab3_1_new), 'new data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
[text/x-patch] v3-0001-pgoutput-refactor-publication-cache-construction.patch (18.1K, ../../[email protected]/3-v3-0001-pgoutput-refactor-publication-cache-construction.patch)
download | inline diff:
From 7e68f96af6e1811113054b4f238b5ed429f7d628 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Fri, 7 Apr 2023 09:55:27 -0700
Subject: [PATCH v3 1/2] pgoutput: refactor publication cache construction
Breaking this logic into a standalone helper should help expose the
behavior for testing. Additionally, move the implementation under the
pg_publication catalog helpers, since it seems like it's inherent to the
publication settings (and it must match the behavior of the initial
sync, which doesn't seem to be controlled by the output plugin at all).
---
src/backend/catalog/pg_publication.c | 215 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 137 +------------
src/include/catalog/pg_proc.dat | 8 +
src/include/catalog/pg_publication.h | 4 +
src/test/regress/expected/publication.out | 26 +++
src/test/regress/sql/publication.sql | 15 ++
6 files changed, 272 insertions(+), 133 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index c488b6370b..df9abf09c3 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1259,3 +1259,218 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+List *
+process_relation_publications(Oid relid, const List *publications,
+ PublicationActions *pubactions,
+ Oid *publish_as_relid)
+{
+ Oid schemaId = get_rel_namespace(relid);
+ List *pubids = GetRelationPublications(relid);
+
+ /*
+ * We don't acquire a lock on the namespace system table as we build
+ * the cache entry using a historic snapshot and all the later changes
+ * are absorbed while decoding WAL.
+ */
+ List *schemaPubids = GetSchemaPublications(schemaId);
+ ListCell *lc;
+ int publish_ancestor_level = 0;
+ bool am_partition = get_rel_relispartition(relid);
+ char relkind = get_rel_relkind(relid);
+ List *rel_publications = NIL;
+
+ *publish_as_relid = relid;
+
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ bool publish = false;
+
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications. Also
+ * track the ancestor level for this publication.
+ */
+ Oid pub_relid = relid;
+ int ancestor_level = 0;
+
+ /*
+ * If this is a FOR ALL TABLES publication, pick the partition
+ * root and set the ancestor level accordingly.
+ */
+ if (pub->alltables)
+ {
+ publish = true;
+ if (pub->pubviaroot && am_partition)
+ {
+ List *ancestors = get_partition_ancestors(relid);
+
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
+ }
+
+ if (!publish)
+ {
+ bool ancestor_published = false;
+
+ /*
+ * For a partition, check if any of the ancestors are
+ * published. If so, note down the topmost ancestor that is
+ * published via this publication, which will be used as the
+ * relation via which to publish the partition's changes.
+ */
+ if (am_partition)
+ {
+ Oid ancestor;
+ int level;
+ List *ancestors = get_partition_ancestors(relid);
+
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
+
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
+ {
+ pub_relid = ancestor;
+ ancestor_level = level;
+ }
+ }
+ }
+
+ if (list_member_oid(pubids, pub->oid) ||
+ list_member_oid(schemaPubids, pub->oid) ||
+ ancestor_published)
+ publish = true;
+ }
+
+ /*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
+ * Don't publish changes for partitioned tables, because
+ * publishing those of its partitions suffices, unless partition
+ * changes won't be published due to pubviaroot being set.
+ */
+ if (publish &&
+ (relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
+ {
+ pubactions->pubinsert |= pub->pubactions.pubinsert;
+ pubactions->pubupdate |= pub->pubactions.pubupdate;
+ pubactions->pubdelete |= pub->pubactions.pubdelete;
+ pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we need to check if the already
+ * calculated level is higher than the new one. If yes, we can
+ * ignore the new value (as it's a child). Otherwise the new
+ * value is an ancestor, so we keep it.
+ */
+ if (publish_ancestor_level > ancestor_level)
+ continue;
+
+ /*
+ * If we found an ancestor higher up in the tree, discard the
+ * list of publications through which we replicate it, and use
+ * the new ancestor.
+ */
+ if (publish_ancestor_level < ancestor_level)
+ {
+ *publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
+
+ /* reset the publication list for this relation */
+ rel_publications = NIL;
+ }
+ else
+ {
+ /* Same ancestor level, has to be the same OID. */
+ Assert(*publish_as_relid == pub_relid);
+ }
+
+ /* Track publications for this ancestor. */
+ rel_publications = lappend(rel_publications, pub);
+ }
+ }
+
+ list_free(pubids);
+ list_free(schemaPubids);
+
+ return rel_publications;
+}
+
+Datum
+pg_get_relation_publishing_info(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ List *publications = NIL;
+ ArrayType *arr;
+ Datum *elems;
+ int nelems,
+ i;
+ TupleDesc tupdesc;
+ HeapTuple htup;
+
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ arr = PG_GETARG_ARRAYTYPE_P(1);
+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT, &elems, NULL,
+ &nelems);
+
+ /* Get Oids of tables from each publication. */
+ for (i = 0; i < nelems; i++)
+ {
+ char *pubname = TextDatumGetCString(elems[i]);
+ Publication *pub = GetPublicationByName(pubname, false);
+
+ publications = lappend(publications, pub);
+ }
+
+ {
+ List *rel_publications;
+ PublicationActions pubactions;
+ Oid publish_as_relid;
+ ListCell *lc;
+ Datum *puboids;
+ ArrayType *puboidarray;
+ Datum values[6];
+ bool nulls[6];
+
+ rel_publications =
+ process_relation_publications(relid, publications, &pubactions,
+ &publish_as_relid);
+
+ /* Translate the rel_publications List into an OID array. */
+ puboids = palloc(sizeof(Datum) * list_length(rel_publications));
+
+ foreach(lc, rel_publications)
+ {
+ Publication *pub = lfirst(lc);
+
+ i = foreach_current_index(lc);
+ puboids[i] = ObjectIdGetDatum(pub->oid);
+ }
+
+ puboidarray = construct_array(puboids, list_length(rel_publications),
+ OIDOID, sizeof(Oid), true, TYPALIGN_INT);
+
+ values[0] = PointerGetDatum(puboidarray);
+ values[1] = ObjectIdGetDatum(publish_as_relid);
+ values[2] = BoolGetDatum(pubactions.pubinsert);
+ values[3] = BoolGetDatum(pubactions.pubupdate);
+ values[4] = BoolGetDatum(pubactions.pubdelete);
+ values[5] = BoolGetDatum(pubactions.pubtruncate);
+
+ memset(nulls, false, sizeof(nulls));
+
+ htup = heap_form_tuple(tupdesc, values, nulls);
+ }
+
+ PG_RETURN_DATUM(HeapTupleGetDatum(htup));
+}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index b08ca55041..640676e7dc 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1986,20 +1986,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Validate the entry */
if (!entry->replicate_valid)
{
- Oid schemaId = get_rel_namespace(relid);
- List *pubids = GetRelationPublications(relid);
-
- /*
- * We don't acquire a lock on the namespace system table as we build
- * the cache entry using a historic snapshot and all the later changes
- * are absorbed while decoding WAL.
- */
- List *schemaPubids = GetSchemaPublications(schemaId);
- ListCell *lc;
- Oid publish_as_relid = relid;
- int publish_ancestor_level = 0;
- bool am_partition = get_rel_relispartition(relid);
- char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
/* Reload publications if needed before use. */
@@ -2063,123 +2049,10 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
* but here we only need to consider ones that the subscriber
* requested.
*/
- foreach(lc, data->publications)
- {
- Publication *pub = lfirst(lc);
- bool publish = false;
-
- /*
- * Under what relid should we publish changes in this publication?
- * We'll use the top-most relid across all publications. Also
- * track the ancestor level for this publication.
- */
- Oid pub_relid = relid;
- int ancestor_level = 0;
-
- /*
- * If this is a FOR ALL TABLES publication, pick the partition
- * root and set the ancestor level accordingly.
- */
- if (pub->alltables)
- {
- publish = true;
- if (pub->pubviaroot && am_partition)
- {
- List *ancestors = get_partition_ancestors(relid);
-
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
- }
- }
-
- if (!publish)
- {
- bool ancestor_published = false;
-
- /*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
- * published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
- */
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
-
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
-
- if (ancestor != InvalidOid)
- {
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
- }
- }
-
- if (list_member_oid(pubids, pub->oid) ||
- list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
- publish = true;
- }
-
- /*
- * If the relation is to be published, determine actions to
- * publish, and list of columns, if appropriate.
- *
- * Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
- */
- if (publish &&
- (relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
- {
- entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
- entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
- entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
- entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
-
- /*
- * We want to publish the changes as the top-most ancestor
- * across all publications. So we need to check if the already
- * calculated level is higher than the new one. If yes, we can
- * ignore the new value (as it's a child). Otherwise the new
- * value is an ancestor, so we keep it.
- */
- if (publish_ancestor_level > ancestor_level)
- continue;
-
- /*
- * If we found an ancestor higher up in the tree, discard the
- * list of publications through which we replicate it, and use
- * the new ancestor.
- */
- if (publish_ancestor_level < ancestor_level)
- {
- publish_as_relid = pub_relid;
- publish_ancestor_level = ancestor_level;
-
- /* reset the publication list for this relation */
- rel_publications = NIL;
- }
- else
- {
- /* Same ancestor level, has to be the same OID. */
- Assert(publish_as_relid == pub_relid);
- }
-
- /* Track publications for this ancestor. */
- rel_publications = lappend(rel_publications, pub);
- }
- }
-
- entry->publish_as_relid = publish_as_relid;
+ rel_publications =
+ process_relation_publications(relid, data->publications,
+ &entry->pubactions,
+ &entry->publish_as_relid);
/*
* Initialize the tuple slot, map, and row filter. These are only used
@@ -2198,8 +2071,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
pgoutput_column_list_init(data, rel_publications, entry);
}
- list_free(pubids);
- list_free(schemaPubids);
list_free(rel_publications);
entry->replicate_valid = true;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6996073989..ed67168374 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11833,6 +11833,14 @@
proname => 'pg_relation_is_publishable', provolatile => 's',
prorettype => 'bool', proargtypes => 'regclass',
prosrc => 'pg_relation_is_publishable' },
+{ oid => '8139',
+ descr => 'get information on how a relation will be published via a list of publications',
+ proname => 'pg_get_relation_publishing_info', provariadic => 'text',
+ provolatile => 's', prorettype => 'record', proargtypes => 'regclass _text',
+ proallargtypes => '{regclass,_text,_oid,oid,bool,bool,bool,bool}',
+ proargmodes => '{i,v,o,o,o,o,o,o}',
+ proargnames => '{relid,pubnames,pubids,pubasrelid,pubinsert,pubupdate,pubdelete,pubtruncate}',
+ prosrc => 'pg_get_relation_publishing_info' },
# rls
{ oid => '3298',
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 6ecaa2a01e..d4e3488917 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -155,4 +155,8 @@ extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
MemoryContext mcxt);
+extern List *process_relation_publications(Oid relid, const List *publications,
+ PublicationActions *pubactions,
+ Oid *publish_as_relid);
+
#endif /* PG_PUBLICATION_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 69dc6cfd85..dd8fc99111 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -5,6 +5,17 @@ CREATE ROLE regress_publication_user LOGIN SUPERUSER;
CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_stream (VARIADIC pubnames text[])
+ RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
+ -- For each publication, show each published root alongside the tables which
+ -- are published via its OID.
+ SELECT p.pubname, rpi.pubasrelid::regclass, pr.prrelid::regclass
+ FROM pg_publication_rel pr
+ JOIN pg_publication p ON (p.oid = pr.prpubid),
+ pg_get_relation_publishing_info(pr.prrelid, VARIADIC pubnames) rpi
+ WHERE p.pubname = ANY (pubnames)
+ ORDER BY p.oid, 2, 3;
+$$ LANGUAGE sql;
-- suppress warning that depends on wal_level
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_default;
@@ -1686,6 +1697,13 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+SELECT * FROM published_stream('pub');
+ pubname | published | synced
+---------+-----------+-----------------
+ pub | sch1.tbl1 | sch1.tbl1
+ pub | sch1.tbl1 | sch2.tbl1_part1
+(2 rows)
+
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
@@ -1712,6 +1730,13 @@ SELECT * FROM pg_publication_tables;
pub | sch2 | tbl1_part1 | {a} |
(1 row)
+SELECT * FROM published_stream('pub');
+ pubname | published | synced
+---------+-----------------+-----------------
+ pub | sch1.tbl1 | sch1.tbl1
+ pub | sch2.tbl1_part1 | sch2.tbl1_part1
+(2 rows)
+
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
DROP TABLE sch1.tbl1;
@@ -1732,6 +1757,7 @@ DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
+DROP FUNCTION published_stream;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
DROP ROLE regress_publication_user_dummy;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index d5051a5e74..6bf9554d48 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -6,6 +6,18 @@ CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_stream (VARIADIC pubnames text[])
+ RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
+ -- For each publication, show each published root alongside the tables which
+ -- are published via its OID.
+ SELECT p.pubname, rpi.pubasrelid::regclass, pr.prrelid::regclass
+ FROM pg_publication_rel pr
+ JOIN pg_publication p ON (p.oid = pr.prpubid),
+ pg_get_relation_publishing_info(pr.prrelid, VARIADIC pubnames) rpi
+ WHERE p.pubname = ANY (pubnames)
+ ORDER BY p.oid, 2, 3;
+$$ LANGUAGE sql;
+
-- suppress warning that depends on wal_level
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_default;
@@ -1064,6 +1076,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
@@ -1078,6 +1091,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
@@ -1096,6 +1110,7 @@ DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
+DROP FUNCTION published_stream;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
--
2.25.1
[text/x-patch] v3-0002-WIP-introduce-publish_via_parent-for-logical-repl.patch (83.2K, ../../[email protected]/4-v3-0002-WIP-introduce-publish_via_parent-for-logical-repl.patch)
download | inline diff:
From a48563919e0d4a70b7115551a5d061b2e17b5243 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Mon, 26 Sep 2022 13:23:51 -0700
Subject: [PATCH v3 2/2] WIP: introduce publish_via_parent for logical
replication
This new relation option allows regular inherited tables to be published
via their root table, just like partitions. This works by hijacking
pg_inherit's inhseqno column, and replacing a (single) existing entry
for the child with the value zero, indicating that it should be treated
as a logical partition by the publication machinery.
Initial sync works by asking the publisher for a list of logical
descendants of the published table, then COPYing them one-by-one into
the root. The publisher reuses the existing pubviaroot logic, adding the
new logical roots to code that previously looked only for partition
roots.
Known bugs/TODOs:
- If two publications publish the same leaves of a table hierarchy, but
have different roots, the shared leaf tables will be incorrectly
duplicated when subscribing to both publications simultaneously.
- I haven't given any thought to interactions with row filters, or to
column lists.
- I'm not sure that I'm taking all the necessary locks yet, and those I
do take may be taken in the wrong order.
---
src/backend/access/common/reloptions.c | 11 +
src/backend/catalog/pg_inherits.c | 123 +++++++-
src/backend/catalog/pg_publication.c | 284 +++++++++++++-----
src/backend/commands/publicationcmds.c | 10 +
src/backend/commands/tablecmds.c | 161 +++++++++-
src/backend/partitioning/partdesc.c | 3 +-
src/backend/replication/logical/tablesync.c | 257 ++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 1 -
src/bin/pg_dump/t/002_pg_dump.pl | 24 ++
src/bin/psql/tab-complete.c | 1 +
src/include/catalog/pg_inherits.h | 8 +-
src/include/catalog/pg_proc.dat | 7 +
src/include/utils/rel.h | 12 +
src/test/regress/expected/publication.out | 315 ++++++++++++++++++++
src/test/regress/sql/publication.sql | 140 +++++++++
src/test/subscription/t/013_partition.pl | 312 +++++++++++++++++++
16 files changed, 1490 insertions(+), 179 deletions(-)
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 11cc431677..6d01e2477d 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -122,6 +122,15 @@ static relopt_bool boolRelOpts[] =
},
false
},
+ {
+ {
+ "publish_via_parent",
+ "Replicate this table's contents via its parent table when publishing with publish_via_partition_root",
+ RELOPT_KIND_HEAP,
+ AccessExclusiveLock
+ },
+ false
+ },
{
{
"fastupdate",
@@ -1877,6 +1886,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind)
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, analyze_scale_factor)},
{"user_catalog_table", RELOPT_TYPE_BOOL,
offsetof(StdRdOptions, user_catalog_table)},
+ {"publish_via_parent", RELOPT_TYPE_BOOL,
+ offsetof(StdRdOptions, publish_via_parent)},
{"parallel_workers", RELOPT_TYPE_INT,
offsetof(StdRdOptions, parallel_workers)},
{"vacuum_index_cleanup", RELOPT_TYPE_ENUM,
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index da969bd2f9..cf90769457 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -24,14 +24,22 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_inherits.h"
+#include "catalog/partition.h"
+#include "miscadmin.h"
#include "parser/parse_type.h"
#include "storage/lmgr.h"
+#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/fmgrprotos.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+static List * find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only);
+
/*
* Entry of a hash table used in find_all_inheritors. See below.
*/
@@ -59,14 +67,14 @@ List *
find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
{
return find_inheritance_children_extended(parentrelId, true, lockmode,
- NULL, NULL);
+ NULL, NULL, false);
}
/*
* find_inheritance_children_extended
*
* As find_inheritance_children, with more options regarding detached
- * partitions.
+ * partitions and logical roots.
*
* If a partition's pg_inherits row is marked "detach pending",
* *detached_exist (if not null) is set true.
@@ -78,16 +86,21 @@ find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
* whether the transaction that marked those partitions as detached appears
* committed to the active snapshot. In addition, *detached_xmin (if not null)
* is set to the xmin of the row of the detached partition.
+ *
+ * If logical_only is true, only tables marked explicitly via publish_via_parent
+ * are included in the output list.
*/
List *
find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
LOCKMODE lockmode, bool *detached_exist,
- TransactionId *detached_xmin)
+ TransactionId *detached_xmin,
+ bool logical_only)
{
List *list = NIL;
Relation relation;
+ Oid index;
SysScanDesc scan;
- ScanKeyData key[1];
+ ScanKeyData key[2];
HeapTuple inheritsTuple;
Oid inhrelid;
Oid *oidarr;
@@ -110,14 +123,24 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
numoids = 0;
relation = table_open(InheritsRelationId, AccessShareLock);
+ index = InheritsParentIndexId;
ScanKeyInit(&key[0],
Anum_pg_inherits_inhparent,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(parentrelId));
- scan = systable_beginscan(relation, InheritsParentIndexId, true,
- NULL, 1, key);
+ if (logical_only)
+ {
+ index = InheritsParentSeqnoIndexId;
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+ }
+
+ scan = systable_beginscan(relation, index, true,
+ NULL, logical_only ? 2 : 1, key);
while ((inheritsTuple = systable_getnext(scan)) != NULL)
{
@@ -254,6 +277,20 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
*/
List *
find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
+{
+ return find_all_inheritors_internal(parentrelId, lockmode, numparents,
+ false);
+}
+
+List *
+find_all_logical_inheritors(Oid parentrelId)
+{
+ return find_all_inheritors_internal(parentrelId, NoLock, NULL, true);
+}
+
+static List *
+find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only)
{
/* hash table for O(1) rel_oid -> rel_numparents cell lookup */
HTAB *seen_rels;
@@ -290,7 +327,9 @@ find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
ListCell *lc;
/* Get the direct children of this rel */
- currentchildren = find_inheritance_children(currentrel, lockmode);
+ currentchildren =
+ find_inheritance_children_extended(currentrel, true, lockmode,
+ NULL, NULL, logical_only);
/*
* Add to the queue only those children not already seen. This avoids
@@ -655,3 +694,73 @@ PartitionHasPendingDetach(Oid partoid)
elog(ERROR, "relation %u is not a partition", partoid);
return false; /* keep compiler quiet */
}
+
+static Oid
+get_logical_parent_worker(Relation inhRel, Oid relid)
+{
+ SysScanDesc scan;
+ ScanKeyData key[2];
+ Oid result = InvalidOid;
+ HeapTuple tuple;
+
+ ScanKeyInit(&key[0],
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 2, key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ Form_pg_inherits form = (Form_pg_inherits) GETSTRUCT(tuple);
+ result = form->inhparent;
+ }
+
+ systable_endscan(scan);
+
+ return result;
+}
+
+static void
+get_logical_ancestors_worker(Relation inhRel, Oid relid, List **ancestors)
+{
+ Oid parentOid;
+
+ /*
+ * Recursion ends at the topmost level, ie., when there's no parent.
+ */
+ parentOid = get_logical_parent_worker(inhRel, relid);
+ if (parentOid == InvalidOid)
+ return;
+
+ *ancestors = lappend_oid(*ancestors, parentOid);
+ get_logical_ancestors_worker(inhRel, parentOid, ancestors);
+}
+
+List *
+get_logical_ancestors(Oid relid, bool is_partition)
+{
+ List *result = NIL;
+ Relation inhRel;
+
+ /* For partitions, this is identical to get_partition_ancestors(). */
+ if (is_partition)
+ return get_partition_ancestors(relid);
+
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
+ get_logical_ancestors_worker(inhRel, relid, &result);
+ table_close(inhRel, AccessShareLock);
+
+ return result;
+}
+
+bool
+has_logical_parent(Relation inhRel, Oid relid)
+{
+ return (get_logical_parent_worker(inhRel, relid) != InvalidOid);
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index df9abf09c3..873cbf955e 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -51,6 +51,7 @@ typedef struct
Oid relid; /* OID of published table */
Oid pubid; /* OID of publication that publishes this
* table. */
+ bool viaroot; /* does pubid use publish_via_partition_root? */
} published_rel;
static void publication_translate_columns(Relation targetrel, List *columns,
@@ -180,56 +181,46 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
}
/*
- * Returns true if the ancestor is in the list of published relations.
- * Otherwise, returns false.
- */
-static bool
-is_ancestor_member_tableinfos(Oid ancestor, List *table_infos)
-{
- ListCell *lc;
-
- foreach(lc, table_infos)
- {
- Oid relid = ((published_rel *) lfirst(lc))->relid;
-
- if (relid == ancestor)
- return true;
- }
-
- return false;
-}
-
-/*
- * Filter out the partitions whose parent tables are also present in the list.
+ * Filter out the tables who are already being published via a logical root.
+ *
+ * If we only had partitioned tables to check, this would be much easier: it's
+ * impossible for a partition root to be in this list unless pubviaroot=true for
+ * that table, so we would just have to check for the existence of an ancestor,
+ * and if any were found then we'd filter the table.
+ *
+ * Logical roots add another wrinkle, because it's acceptable for a
+ * pubviaroot=false publication to contain a logical root. (Logical roots can
+ * contain data of their own, unlike partition roots.) It's also possible for a
+ * logical root to be included in a pubviaroot publication without any or all of
+ * its children. This is not possible for a partition root, where all of the
+ * leaves are implicitly included.
+ *
+ * So, to keep things relatively consistent, this now uses the same code that
+ * the output plugin uses to decide which relation OID to publish data under. If
+ * the publishing OID is not the same as the relation OID, we remove it from
+ * this list.
+ *
+ * XXX And that's expensive.
*/
static void
-filter_partitions(List *table_infos)
+filter_published_descendants(List *table_infos, List *publications)
{
ListCell *lc;
foreach(lc, table_infos)
{
- bool skip = false;
- List *ancestors = NIL;
- ListCell *lc2;
published_rel *table_info = (published_rel *) lfirst(lc);
+ Oid publish_as_relid;
+ List *rel_publications;
- if (get_rel_relispartition(table_info->relid))
- ancestors = get_partition_ancestors(table_info->relid);
-
- foreach(lc2, ancestors)
- {
- Oid ancestor = lfirst_oid(lc2);
-
- if (is_ancestor_member_tableinfos(ancestor, table_infos))
- {
- skip = true;
- break;
- }
- }
+ rel_publications =
+ process_relation_publications(table_info->relid, publications, NULL,
+ &publish_as_relid);
- if (skip)
+ if (publish_as_relid != table_info->relid)
table_infos = foreach_delete_current(table_infos, lc);
+
+ list_free(rel_publications);
}
}
@@ -805,11 +796,15 @@ List *
GetAllTablesPublicationRelations(bool pubviaroot)
{
Relation classRel;
+ Relation inhRel;
ScanKeyData key[1];
TableScanDesc scan;
HeapTuple tuple;
List *result = NIL;
+ /* TODO: is there a required order to acquire these locks? */
+ if (pubviaroot)
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
classRel = table_open(RelationRelationId, AccessShareLock);
ScanKeyInit(&key[0],
@@ -825,7 +820,8 @@ GetAllTablesPublicationRelations(bool pubviaroot)
Oid relid = relForm->oid;
if (is_publishable_class(relid, relForm) &&
- !(relForm->relispartition && pubviaroot))
+ !(relForm->relispartition && pubviaroot) &&
+ !(pubviaroot && has_logical_parent(inhRel, relid)))
result = lappend_oid(result, relid);
}
@@ -854,6 +850,9 @@ GetAllTablesPublicationRelations(bool pubviaroot)
}
table_close(classRel, AccessShareLock);
+ if (pubviaroot)
+ table_close(inhRel, AccessShareLock);
+
return result;
}
@@ -1070,6 +1069,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
int nelems,
i;
bool viaroot = false;
+ List *publications = NIL;
/* create a function context for cross-call persistence */
funcctx = SRF_FIRSTCALL_INIT();
@@ -1093,6 +1093,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
ListCell *lc;
pub_elem = GetPublicationByName(TextDatumGetCString(elems[i]), false);
+ publications = lappend(publications, pub_elem);
/*
* Publications support partitioned tables. If
@@ -1132,6 +1133,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
table_info->relid = lfirst_oid(lc);
table_info->pubid = pub_elem->oid;
+ table_info->viaroot = pub_elem->pubviaroot;
table_infos = lappend(table_infos, table_info);
}
@@ -1141,15 +1143,14 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
}
/*
- * If the publication publishes partition changes via their respective
- * root partitioned tables, we must exclude partitions in favor of
- * including the root partitioned tables. Otherwise, the function
- * could return both the child and parent tables which could cause
- * data of the child table to be double-published on the subscriber
- * side.
+ * If the publication publishes table changes via their respective
+ * logical root tables, we must exclude partitions in favor of
+ * including the root tables. Otherwise, the function could return
+ * both the child and parent tables which could cause data of the
+ * child table to be double-published on the subscriber side.
*/
if (viaroot)
- filter_partitions(table_infos);
+ filter_published_descendants(table_infos, publications);
/* Construct a tuple descriptor for the result rows. */
tupdesc = CreateTemplateTupleDesc(NUM_PUBLICATION_TABLES_ELEM);
@@ -1165,6 +1166,8 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
funcctx->tuple_desc = BlessTupleDesc(tupdesc);
funcctx->user_fctx = (void *) table_infos;
+ list_free(publications);
+
MemoryContextSwitchTo(oldcontext);
}
@@ -1260,6 +1263,118 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+/*
+ * Returns a list of tables that should be copied during initial sync for the
+ * given root table and publications.
+ *
+ * The only time this list consists of anything more than the table itself is
+ * when a publication's publish_via_partition_root is set to true and the table
+ * has inherited child tables in the publications that have been marked with
+ * publish_via_parent.
+ */
+Datum
+pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ List *tables = NIL;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ Oid rootid = PG_GETARG_OID(0);
+ MemoryContext oldcontext;
+ ArrayType *arr;
+ Datum *elems;
+ int nelems,
+ i;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ /*
+ * Deconstruct the parameter into elements where each element is a
+ * publication name.
+ */
+ arr = PG_GETARG_ARRAYTYPE_P(1);
+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT,
+ &elems, NULL, &nelems);
+
+ /* Get Oids of tables from each publication. */
+ for (i = 0; i < nelems; i++)
+ {
+ char *pubname = TextDatumGetCString(elems[i]);
+ Publication *publication = GetPublicationByName(pubname, false);
+ List *pub_tables = NIL;
+
+ /* TODO: do the tables in this list need to be locked? */
+ if (publication->pubviaroot)
+ pub_tables = find_all_logical_inheritors(rootid);
+ else
+ pub_tables = list_make1_oid(rootid);
+
+ if (!publication->alltables)
+ {
+ List *relids;
+ List *schemarelids;
+ List *published;
+ ListCell *cell;
+
+ relids = GetPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_ALL);
+ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_LEAF);
+ published = list_concat_unique_oid(relids, schemarelids);
+
+ /*
+ * First we have to check to make sure the root table is
+ * actually part of the publication; if not, none of its
+ * descendants' contents belong to it.
+ */
+ if (!list_member_oid(published, rootid))
+ continue;
+
+ /*
+ * Now filter out any descendants that aren't part of this
+ * particular publication.
+ */
+ foreach(cell, pub_tables)
+ {
+ Oid current = lfirst_oid(cell);
+
+ if (!list_member_oid(published, current))
+ pub_tables = foreach_delete_current(pub_tables, cell);
+ }
+ }
+
+ tables = list_concat_unique_oid(tables, pub_tables);
+ }
+
+ funcctx->user_fctx = (void *) tables;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+ tables = (List *) funcctx->user_fctx;
+
+ if (funcctx->call_cntr < list_length(tables))
+ {
+ Oid relid = list_nth_oid(tables, funcctx->call_cntr);
+
+ SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ }
+
+ SRF_RETURN_DONE(funcctx);
+}
+
List *
process_relation_publications(Oid relid, const List *publications,
PublicationActions *pubactions,
@@ -1279,8 +1394,7 @@ process_relation_publications(Oid relid, const List *publications,
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
-
- *publish_as_relid = relid;
+ Oid publish_candidate = relid;
foreach(lc, publications)
{
@@ -1296,55 +1410,57 @@ process_relation_publications(Oid relid, const List *publications,
int ancestor_level = 0;
/*
- * If this is a FOR ALL TABLES publication, pick the partition
+ * If this is a FOR ALL TABLES publication, pick the logical
* root and set the ancestor level accordingly.
*/
if (pub->alltables)
{
publish = true;
- if (pub->pubviaroot && am_partition)
+ if (pub->pubviaroot)
{
- List *ancestors = get_partition_ancestors(relid);
+ List *ancestors;
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ if (ancestors != NIL)
+ {
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
}
if (!publish)
{
bool ancestor_published = false;
+ Oid ancestor;
+ int level;
+ List *ancestors;
/*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
+ * Check if any of the logical ancestors (that is, partition
+ * parents or tables marked with publish_via_parent) are
+ * published. If so, note down the topmost ancestor that is
* published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
+ * relation via which to publish this table's changes.
*/
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
-
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
- if (ancestor != InvalidOid)
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
{
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
+ pub_relid = ancestor;
+ ancestor_level = level;
}
}
if (list_member_oid(pubids, pub->oid) ||
list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
+ (am_partition && ancestor_published))
publish = true;
}
@@ -1359,10 +1475,13 @@ process_relation_publications(Oid relid, const List *publications,
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
- pubactions->pubinsert |= pub->pubactions.pubinsert;
- pubactions->pubupdate |= pub->pubactions.pubupdate;
- pubactions->pubdelete |= pub->pubactions.pubdelete;
- pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+ if (pubactions)
+ {
+ pubactions->pubinsert |= pub->pubactions.pubinsert;
+ pubactions->pubupdate |= pub->pubactions.pubupdate;
+ pubactions->pubdelete |= pub->pubactions.pubdelete;
+ pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+ }
/*
* We want to publish the changes as the top-most ancestor
@@ -1381,7 +1500,7 @@ process_relation_publications(Oid relid, const List *publications,
*/
if (publish_ancestor_level < ancestor_level)
{
- *publish_as_relid = pub_relid;
+ publish_candidate = pub_relid;
publish_ancestor_level = ancestor_level;
/* reset the publication list for this relation */
@@ -1390,7 +1509,7 @@ process_relation_publications(Oid relid, const List *publications,
else
{
/* Same ancestor level, has to be the same OID. */
- Assert(*publish_as_relid == pub_relid);
+ Assert(publish_candidate == pub_relid);
}
/* Track publications for this ancestor. */
@@ -1401,6 +1520,9 @@ process_relation_publications(Oid relid, const List *publications,
list_free(pubids);
list_free(schemaPubids);
+ if (publish_as_relid)
+ *publish_as_relid = publish_candidate;
+
return rel_publications;
}
@@ -1434,7 +1556,7 @@ pg_get_relation_publishing_info(PG_FUNCTION_ARGS)
{
List *rel_publications;
- PublicationActions pubactions;
+ PublicationActions pubactions = {0};
Oid publish_as_relid;
ListCell *lc;
Datum *puboids;
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f4ba572697..4cabc9719e 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -238,6 +238,8 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* parent table, but the bitmap contains the replica identity
* information of the child table. So, get the column number of the
* child table as parent and child column order could be different.
+ *
+ * TODO: is this applicable to publish_via_parent?
*/
if (context->pubviaroot)
{
@@ -286,6 +288,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
*
* Note that even though the row filter used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
+ *
+ * TODO: is this applicable to publish_via_parent?
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
@@ -336,6 +340,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
* the column list.
*
* Returns true if any replica identity column is not covered by column list.
+ *
+ * TODO: publish_via_parent?
*/
bool
pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
@@ -628,6 +634,8 @@ TransformPubWhereClauses(List *tables, const char *queryString,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
+ * TODO: decide how this interacts with publish_via_parent
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ -715,6 +723,8 @@ CheckPubRelationColumnList(char *pubname, List *tables,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
+ * TODO: decide if this interacts with publish_via_parent
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index d985278ac6..8250640643 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -355,7 +355,8 @@ static bool MergeCheckConstraint(List *constraints, char *name, Node *expr);
static void MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel);
static void MergeConstraintsIntoExisting(Relation child_rel, Relation parent_rel);
static void StoreCatalogInheritance(Oid relationId, List *supers,
- bool child_is_partition);
+ bool child_is_partition,
+ bool publish_via_parent);
static void StoreCatalogInheritance1(Oid relationId, Oid parentOid,
int32 seqNumber, Relation inhRelation,
bool child_is_partition);
@@ -577,6 +578,7 @@ static void ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel,
const char *tablespacename, LOCKMODE lockmode);
static void ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode);
static void ATExecSetTableSpaceNoStorage(Relation rel, Oid newTableSpace);
+static void change_publish_via_parent(Relation rel, bool set);
static void ATExecSetRelOptions(Relation rel, List *defList,
AlterTableType operation,
LOCKMODE lockmode);
@@ -1115,7 +1117,8 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
}
/* Store inheritance information for new rel. */
- StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL);
+ StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL,
+ RelationIsPublishedViaParent(rel));
/*
* Process the partitioning specification (if any) and store the partition
@@ -3218,7 +3221,7 @@ MergeCheckConstraint(List *constraints, char *name, Node *expr)
*/
static void
StoreCatalogInheritance(Oid relationId, List *supers,
- bool child_is_partition)
+ bool child_is_partition, bool publish_via_parent)
{
Relation relation;
int32 seqNumber;
@@ -3229,6 +3232,22 @@ StoreCatalogInheritance(Oid relationId, List *supers,
*/
Assert(OidIsValid(relationId));
+ /*
+ * publish_via_parent requires exactly one parent. Try to keep this
+ * messaging consistent with ALTER TABLE/change_publish_via_parent().
+ *
+ * Earlier, in MergeAttributes, we already checked that we had ownership of
+ * the parent table, so we don't need to check that again the way that
+ * change_publish_via_parent() does.
+ */
+ if (publish_via_parent && list_length(supers) != 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg((list_length(supers) == 0)
+ ? "table \"%s\" does not inherit from any tables"
+ : "table \"%s\" inherits from multiple tables",
+ get_rel_name(relationId))));
+
if (supers == NIL)
return;
@@ -3243,7 +3262,12 @@ StoreCatalogInheritance(Oid relationId, List *supers,
*/
relation = table_open(InheritsRelationId, RowExclusiveLock);
- seqNumber = 1;
+ /*
+ * Normally inhseqno starts at one, but for publish_via_parent, it's given a
+ * sentinel value of zero. (In that case, we'll only go through this loop
+ * once, as the list_length() check above enforces.)
+ */
+ seqNumber = publish_via_parent ? 0 : 1;
foreach(entry, supers)
{
Oid parentOid = lfirst_oid(entry);
@@ -14276,6 +14300,88 @@ ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel, const char *tablespacen
tab->newTableSpace = tablespaceId;
}
+static void
+change_publish_via_parent(Relation rel, bool set)
+{
+ Relation inhRel;
+ ScanKeyData key;
+ SysScanDesc scan;
+ Oid parentOid;
+ Relation parentRel;
+ HeapTuple tuple, copyTuple;
+ Form_pg_inherits form;
+
+ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
+ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
+
+ /*
+ * We have to make sure that an inheritance relationship already exists, and
+ * that there is only one candidate parent for this table.
+ *
+ * TODO: do we have to lock the tables themselves to avoid races?
+ */
+ ScanKeyInit(&key,
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationGetRelid(rel)));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 1, &key);
+ tuple = systable_getnext(scan);
+
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" does not inherit from any tables",
+ RelationGetRelationName(rel))));
+
+ form = (Form_pg_inherits) GETSTRUCT(tuple);
+ parentOid = form->inhparent;
+ copyTuple = heap_copytuple(tuple);
+
+ if (HeapTupleIsValid(systable_getnext(scan)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" inherits from multiple tables",
+ RelationGetRelationName(rel))));
+
+ systable_endscan(scan);
+
+ if (set)
+ {
+ /*
+ * Open our parent table and check permissions. Like ATTACH PARTITION,
+ * we require ownership of the parent table in addition to the child
+ * (which has already been checked in ATPrepCmd).
+ *
+ * TODO: what lock strength is needed here? Is nesting it inside the
+ * inhRel lock a risk?
+ */
+ parentRel = table_open(parentOid, AccessExclusiveLock);
+ ATSimplePermissions(AT_SetRelOptions, parentRel, ATT_TABLE);
+ table_close(parentRel, NoLock); /* TODO: do we need to hang onto the lock? */
+ }
+ else
+ {
+ /*
+ * Ownership of the parent isn't needed for unsetting the flag, just
+ * like we wouldn't check ownership during NO INHERIT.
+ */
+ }
+
+ /*
+ * Mark the inheritance as a logical root by setting inhseqno to zero.
+ * Unmark it by setting inhseqno to one (since there's only one parent).
+ */
+ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
+ form->inhseqno = set ? 0 : 1;
+
+ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
+
+ heap_freetuple(copyTuple);
+ table_close(inhRel, RowExclusiveLock);
+}
+
/*
* Set, reset, or replace reloptions.
*/
@@ -14387,6 +14493,39 @@ ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation,
}
}
+ /* Extra work for publish_via_parent */
+ if (rel->rd_rel->relkind == RELKIND_RELATION)
+ {
+ List *heap_options = untransformRelOptions(newOptions);
+ ListCell *cell;
+ bool curSetting = RelationIsPublishedViaParent(rel);
+ DefElem *pubElem = NULL;
+
+ foreach (cell, heap_options)
+ {
+ DefElem *defel = lfirst_node(DefElem, cell);
+
+ if (strcmp(defel->defname, "publish_via_parent") == 0)
+ {
+ pubElem = defel;
+ break;
+ }
+ }
+
+ if (pubElem)
+ {
+ bool newSetting = defGetBoolean(pubElem);
+
+ if (newSetting != curSetting)
+ change_publish_via_parent(rel, newSetting);
+ }
+ else if (curSetting)
+ {
+ /* publish_via_parent was RESET */
+ change_publish_via_parent(rel, false);
+ }
+ }
+
/*
* All we need do here is update the pg_class row; the new options will be
* propagated into relcaches during post-commit cache inval.
@@ -14979,6 +15118,13 @@ ATExecAddInherit(Relation child_rel, RangeVar *parent, LOCKMODE lockmode)
trigger_name, RelationGetRelationName(child_rel)),
errdetail("ROW triggers with transition tables are not supported in inheritance hierarchies.")));
+ /* publish_via_parent tables are not allowed to add additional parents. */
+ if (RelationIsPublishedViaParent(child_rel))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
+ "publish_via_parent", RelationGetRelationName(child_rel))));
+
/* OK to create inheritance */
CreateInheritance(child_rel, parent_rel);
@@ -15395,6 +15541,13 @@ ATExecDropInherit(Relation rel, RangeVar *parent, LOCKMODE lockmode)
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot change inheritance of a partition")));
+ /* publish_via_parent tables are not allowed to remove their parent. */
+ if (RelationIsPublishedViaParent(rel))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
+ "publish_via_parent", RelationGetRelationName(rel))));
+
/*
* AccessShareLock on the parent is probably enough, seeing that DROP
* TABLE doesn't lock parent tables at all. We need some lock since we'll
diff --git a/src/backend/partitioning/partdesc.c b/src/backend/partitioning/partdesc.c
index 7a2b5e57ff..dfb27990f9 100644
--- a/src/backend/partitioning/partdesc.c
+++ b/src/backend/partitioning/partdesc.c
@@ -162,7 +162,8 @@ RelationBuildPartitionDesc(Relation rel, bool omit_detached)
inhoids = find_inheritance_children_extended(RelationGetRelid(rel),
omit_detached, NoLock,
&detached_exist,
- &detached_xmin);
+ &detached_xmin,
+ false);
nparts = list_length(inhoids);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6d461654ab..8d0f081dac 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -758,11 +758,12 @@ copy_read_data(void *outbuf, int minread, int maxread)
/*
* Get information about remote relation in similar fashion the RELATION
* message provides during replication. This function also returns the relation
- * qualifications to be used in the COPY command.
+ * qualifications to be used in the COPY command, and the list of tables to COPY
+ * (which for most tables will contain only one entry).
*/
static void
fetch_remote_table_info(char *nspname, char *relname,
- LogicalRepRelation *lrel, List **qual)
+ LogicalRepRelation *lrel, List **qual, List **to_copy)
{
WalRcvExecResult *res;
StringInfoData cmd;
@@ -1073,6 +1074,87 @@ fetch_remote_table_info(char *nspname, char *relname,
walrcv_clear_result(res);
}
+ /*
+ * See if there are any other tables to be copied besides the original. This
+ * happens when a descendant in the inheritance relationship is marked with
+ * publish_via_parent and is part of a publication with
+ * publish_via_partition_root = true.
+ *
+ * FIXME: if two publications have conflicting settings for
+ * publish_via_partition_root, this behaves strangely. It looks like that's
+ * true for regular partitions too...
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000)
+ {
+ Oid descRow[] = {TEXTOID, TEXTOID};
+ StringInfoData pub_names;
+
+ /* Build the pubname list. */
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ char *pubname = strVal(lfirst(lc));
+
+ if (foreach_current_index(lc) > 0)
+ appendStringInfoString(&pub_names, ", ");
+
+ appendStringInfoString(&pub_names, quote_literal_cstr(pubname));
+ }
+
+ /*
+ * Ask the server what it wants us to sync.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT n.nspname, c.relname"
+ " FROM pg_catalog.pg_publication p,"
+ " pg_catalog.pg_class c"
+ " JOIN pg_catalog.pg_namespace n"
+ " ON (c.relnamespace = n.oid)"
+ " JOIN LATERAL pg_catalog.pg_get_publication_rels_to_sync(%u::regclass, p.pubname) relid"
+ " ON (c.oid = relid)"
+ " WHERE p.pubname IN ( %s )",
+ lrel->remoteid, pub_names.data);
+
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(descRow), descRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch logical descendants for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *desc_nspname;
+ char *desc_relname;
+ char *quoted;
+
+ desc_nspname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+ desc_relname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ quoted = quote_qualified_identifier(desc_nspname, desc_relname);
+ *to_copy = lappend(*to_copy, quoted);
+
+ ExecClearTuple(slot);
+ }
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+
+ pfree(pub_names.data);
+ }
+ else
+ {
+ /* For older servers, we only COPY the table itself. */
+ char *quoted = quote_qualified_identifier(lrel->nspname,
+ lrel->relname);
+ *to_copy = lappend(*to_copy, quoted);
+ }
+
pfree(cmd.data);
}
@@ -1087,6 +1169,8 @@ copy_table(Relation rel)
LogicalRepRelMapEntry *relmapentry;
LogicalRepRelation lrel;
List *qual = NIL;
+ List *to_copy = NIL;
+ ListCell *cur;
WalRcvExecResult *res;
StringInfoData cmd;
CopyFromState cstate;
@@ -1096,7 +1180,8 @@ copy_table(Relation rel)
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &lrel, &qual,
+ &to_copy);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1105,105 +1190,109 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
- /* Start copy on the publisher. */
- initStringInfo(&cmd);
-
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ foreach(cur, to_copy)
{
- appendStringInfo(&cmd, "COPY %s (",
- quote_qualified_identifier(lrel.nspname, lrel.relname));
-
- /*
- * XXX Do we need to list the columns in all cases? Maybe we're
- * replicating all columns?
- */
- for (int i = 0; i < lrel.natts; i++)
- {
- if (i > 0)
- appendStringInfoString(&cmd, ", ");
+ char *quoted_name = lfirst(cur);
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- }
+ /* Start copy on the publisher. */
+ initStringInfo(&cmd);
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
- else
- {
- /*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
- */
- appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ /* Regular table with no row filter */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
- appendStringInfoString(&cmd, ", ");
- }
+ appendStringInfo(&cmd, "COPY %s (", quoted_name);
- appendStringInfoString(&cmd, " FROM ");
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're
+ * replicating all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
- /*
- * For regular tables, make sure we don't copy data from a child that
- * inherits the named table as those will be copied separately.
- */
- if (lrel.relkind == RELKIND_RELATION)
- appendStringInfoString(&cmd, "ONLY ");
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
- appendStringInfoString(&cmd, quote_qualified_identifier(lrel.nspname, lrel.relname));
- /* list of OR'ed filters */
- if (qual != NIL)
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
+ else
{
- ListCell *lc;
- char *q = strVal(linitial(qual));
+ /*
+ * For non-tables and tables with row filters, we need to do COPY
+ * (SELECT ...), but we can't just do SELECT * because we need to not
+ * copy generated columns. For tables with any row filters, build a
+ * SELECT query with OR'ed row filters for COPY.
+ */
+ appendStringInfoString(&cmd, "COPY (SELECT ");
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+
+ appendStringInfoString(&cmd, " FROM ");
+
+ /*
+ * For regular tables, make sure we don't copy data from a child that
+ * inherits the named table as those will be copied separately.
+ */
+ if (lrel.relkind == RELKIND_RELATION)
+ appendStringInfoString(&cmd, "ONLY ");
- appendStringInfo(&cmd, " WHERE %s", q);
- for_each_from(lc, qual, 1)
+ appendStringInfoString(&cmd, quoted_name);
+ /* list of OR'ed filters */
+ if (qual != NIL)
{
- q = strVal(lfirst(lc));
- appendStringInfo(&cmd, " OR %s", q);
+ ListCell *lc;
+ char *q = strVal(linitial(qual));
+
+ appendStringInfo(&cmd, " WHERE %s", q);
+ for_each_from(lc, qual, 1)
+ {
+ q = strVal(lfirst(lc));
+ appendStringInfo(&cmd, " OR %s", q);
+ }
+ list_free_deep(qual);
}
- list_free_deep(qual);
- }
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
- /*
- * Prior to v16, initial table synchronization will use text format even
- * if the binary option is enabled for a subscription.
- */
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000 &&
- MySubscription->binary)
- {
- appendStringInfoString(&cmd, " WITH (FORMAT binary)");
- options = list_make1(makeDefElem("format",
- (Node *) makeString("binary"), -1));
- }
+ /*
+ * Prior to v16, initial table synchronization will use text format even
+ * if the binary option is enabled for a subscription.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000 &&
+ MySubscription->binary)
+ {
+ appendStringInfoString(&cmd, " WITH (FORMAT binary)");
+ options = list_make1(makeDefElem("format",
+ (Node *) makeString("binary"), -1));
+ }
- res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
- pfree(cmd.data);
- if (res->status != WALRCV_OK_COPY_OUT)
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("could not start initial contents copy for table \"%s.%s\": %s",
- lrel.nspname, lrel.relname, res->err)));
- walrcv_clear_result(res);
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
+ pfree(cmd.data);
+ if (res->status != WALRCV_OK_COPY_OUT)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not start initial contents copy for table \"%s.%s\" from remote %s: %s",
+ lrel.nspname, lrel.relname, quoted_name, res->err)));
+ walrcv_clear_result(res);
- copybuf = makeStringInfo();
+ copybuf = makeStringInfo();
- pstate = make_parsestate(NULL);
- (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
- NULL, false, false);
+ pstate = make_parsestate(NULL);
+ (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
+ NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
- cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
+ attnamelist = make_copy_attnamelist(relmapentry);
+ cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
- /* Do the copy */
- (void) CopyFrom(cstate);
+ /* Do the copy */
+ (void) CopyFrom(cstate);
+ }
logicalrep_rel_close(relmapentry, NoLock);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 640676e7dc..7c54b46ac7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1462,7 +1462,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Switch relation if publishing via root. */
if (relentry->publish_as_relid != RelationGetRelid(relation))
{
- Assert(relation->rd_rel->relispartition);
ancestor = RelationIdGetRelation(relentry->publish_as_relid);
targetrel = ancestor;
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 63bb4689d4..d90745af71 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -4573,6 +4573,30 @@ my %tests = (
no_table_access_method => 1,
only_dump_measurement => 1,
},
+ },
+
+ 'CREATE TABLE regress_pg_dump_publish_via_parent' => {
+ create_order => 3,
+ create_sql => '
+ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent (col1 int);
+ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1() INHERITS (dump_test.regress_pg_dump_publish_via_parent) WITH (publish_via_parent);',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent (\E
+ \n\s+\Qcol1 integer\E
+ \n\);
+ (\n.*)+?
+ \n\QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1 (\E
+ \n\)
+ \n\QINHERITS (dump_test.regress_pg_dump_publish_via_parent)\E
+ \n\QWITH (publish_via_parent='true')\E;/xm,
+ like => {
+ %full_runs, %dump_test_schema_runs, section_pre_data => 1,
+ },
+ unlike => {
+ binary_upgrade => 1,
+ exclude_dump_test_schema => 1,
+ only_dump_measurement => 1,
+ },
});
#########################################
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 677847e434..4606272c0c 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1302,6 +1302,7 @@ static const char *const table_storage_parameters[] = {
"fillfactor",
"log_autovacuum_min_duration",
"parallel_workers",
+ "publish_via_parent",
"toast.autovacuum_enabled",
"toast.autovacuum_freeze_max_age",
"toast.autovacuum_freeze_min_age",
diff --git a/src/include/catalog/pg_inherits.h b/src/include/catalog/pg_inherits.h
index ce154ab943..fc539e7401 100644
--- a/src/include/catalog/pg_inherits.h
+++ b/src/include/catalog/pg_inherits.h
@@ -23,6 +23,7 @@
#include "nodes/pg_list.h"
#include "storage/lock.h"
+#include "utils/relcache.h"
/* ----------------
* pg_inherits definition. cpp turns this into
@@ -46,14 +47,17 @@ typedef FormData_pg_inherits *Form_pg_inherits;
DECLARE_UNIQUE_INDEX_PKEY(pg_inherits_relid_seqno_index, 2680, InheritsRelidSeqnoIndexId, on pg_inherits using btree(inhrelid oid_ops, inhseqno int4_ops));
DECLARE_INDEX(pg_inherits_parent_index, 2187, InheritsParentIndexId, on pg_inherits using btree(inhparent oid_ops));
+DECLARE_INDEX(pg_inherits_parent_seqno_index, 8138, InheritsParentSeqnoIndexId, on pg_inherits using btree(inhparent oid_ops, inhseqno int4_ops));
extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);
extern List *find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
- LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin);
+ LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin,
+ bool logical_only);
extern List *find_all_inheritors(Oid parentrelId, LOCKMODE lockmode,
List **numparents);
+extern List *find_all_logical_inheritors(Oid parentrelId);
extern bool has_subclass(Oid relationId);
extern bool has_superclass(Oid relationId);
extern bool typeInheritsFrom(Oid subclassTypeId, Oid superclassTypeId);
@@ -63,5 +67,7 @@ extern bool DeleteInheritsTuple(Oid inhrelid, Oid inhparent,
bool expect_detach_pending,
const char *childname);
extern bool PartitionHasPendingDetach(Oid partoid);
+extern List *get_logical_ancestors(Oid relid, bool is_partition);
+extern bool has_logical_parent(Relation inhRel, Oid relid);
#endif /* PG_INHERITS_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ed67168374..a60c6b66d8 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11833,6 +11833,13 @@
proname => 'pg_relation_is_publishable', provolatile => 's',
prorettype => 'bool', proargtypes => 'regclass',
prosrc => 'pg_relation_is_publishable' },
+{ oid => '8137',
+ descr => 'get the relations to copy from all specified publications during a table\'s initial sync',
+ proname => 'pg_get_publication_rels_to_sync', prorows => '10',
+ provariadic => 'text', proretset => 't', provolatile => 's',
+ prorettype => 'regclass', proargtypes => 'regclass _text',
+ proargmodes => '{i,v}', proargnames => '{rootid,pubnames}',
+ prosrc => 'pg_get_publication_rels_to_sync' },
{ oid => '8139',
descr => 'get information on how a relation will be published via a list of publications',
proname => 'pg_get_relation_publishing_info', provariadic => 'text',
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 1426a353cd..3c5ea5a37a 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -339,6 +339,7 @@ typedef struct StdRdOptions
int toast_tuple_target; /* target for tuple toasting */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
+ bool publish_via_parent; /* publish via parent's relid for logrep */
int parallel_workers; /* max number of parallel workers */
StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */
bool vacuum_truncate; /* enables vacuum to truncate a relation */
@@ -388,6 +389,17 @@ typedef struct StdRdOptions
(relation)->rd_rel->relkind == RELKIND_MATVIEW) ? \
((StdRdOptions *) (relation)->rd_options)->user_catalog_table : false)
+/*
+ * RelationIsPublishedViaParent
+ * Returns whether the relation's contents should be logically replicated
+ * via its parent table's relid, when published under the effects of
+ * publish_via_partition_root. Note multiple eval of argument!
+ */
+#define RelationIsPublishedViaParent(relation) \
+ ((relation)->rd_options && \
+ ((relation)->rd_rel->relkind == RELKIND_RELATION) && \
+ ((StdRdOptions *) (relation)->rd_options)->publish_via_parent)
+
/*
* RelationGetParallelWorkers
* Returns the relation's parallel_workers reloption setting.
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index dd8fc99111..5ee5b8ea3b 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -5,6 +5,17 @@ CREATE ROLE regress_publication_user LOGIN SUPERUSER;
CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_sync (VARIADIC pubnames text[])
+ RETURNS TABLE (published regclass, synced regclass) AS $$
+ -- Show each published table alongside the tables to be copied into it during
+ -- initial sync.
+ SELECT t.published, synced
+ FROM (SELECT DISTINCT relid::regclass AS published
+ FROM pg_get_publication_tables(VARIADIC pubnames)) t
+ JOIN LATERAL pg_get_publication_rels_to_sync(t.published, VARIADIC pubnames) synced
+ ON true
+ ORDER BY 1, 2;
+$$ LANGUAGE sql;
CREATE FUNCTION published_stream (VARIADIC pubnames text[])
RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
-- For each publication, show each published root alongside the tables which
@@ -1697,6 +1708,12 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+SELECT * FROM published_sync('pub');
+ published | synced
+-----------+-----------
+ sch1.tbl1 | sch1.tbl1
+(1 row)
+
SELECT * FROM published_stream('pub');
pubname | published | synced
---------+-----------+-----------------
@@ -1730,6 +1747,12 @@ SELECT * FROM pg_publication_tables;
pub | sch2 | tbl1_part1 | {a} |
(1 row)
+SELECT * FROM published_sync('pub');
+ published | synced
+-----------------+-----------------
+ sch2.tbl1_part1 | sch2.tbl1_part1
+(1 row)
+
SELECT * FROM published_stream('pub');
pubname | published | synced
---------+-----------------+-----------------
@@ -1752,12 +1775,304 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+-- should do nothing at all
+ALTER TABLE sch1.iroot SET (publish_via_parent = false);
+ALTER TABLE sch1.iroot RESET (publish_via_parent);
+-- marking roots between unrelated tables is not allowed
+ALTER TABLE sch1.ipart1 SET (publish_via_parent);
+ERROR: table "ipart1" does not inherit from any tables
+CREATE TABLE fail (a int) WITH (publish_via_parent);
+ERROR: table "fail" does not inherit from any tables
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
+ SET (publish_via_parent);
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
+ SET (publish_via_parent);
+ERROR: table "ipart2" inherits from multiple tables
+-- once publish_via_parent is set, inheritance cannot be changed
+ALTER TABLE sch1.ipart2 SET (publish_via_parent);
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
+ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should fail
+ERROR: must be owner of table part
+RESET SESSION AUTHORIZATION;
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should also fail
+ERROR: must be owner of table root
+CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+ERROR: must be owner of table root
+RESET SESSION AUTHORIZATION;
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
+ WITH (publish_via_partition_root);
+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
+ ONLY sch1.ipart2
+ WITH (publish_via_partition_root);
+-- At this point, the published trees look like this:
+--
+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
+-- ------------------ ----------------------- -----------------------
+-- iroot iroot
+-- +- ipart1 ipart1 |
+-- | +- ipart1_a | +--- ipart1_a
+-- | +- ipart1_a1 +--- ipart1_a1 |
+-- +- ipart2 +- ipart2
+-- What a subscription to only ipub_root should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
+ relid
+----------------
+ sch1.iroot
+ sch1.ipart1
+ sch1.ipart2
+ sch1.ipart1_a
+ sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_sync('ipub_root');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a | sch1.ipart1_a
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_root');
+ pubname | published | synced
+-----------+----------------+----------------
+ ipub_root | sch1.iroot | sch1.iroot
+ ipub_root | sch1.ipart1 | sch1.ipart1
+ ipub_root | sch1.ipart2 | sch1.ipart2
+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
+ ipub_root | sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+-- What a subscription to only ipub_part1 should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
+ relid
+-------------
+ sch1.ipart1
+(1 row)
+
+SELECT * FROM published_sync('ipub_part1');
+ published | synced
+-------------+----------------
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+(2 rows)
+
+SELECT * FROM published_stream('ipub_part1');
+ pubname | published | synced
+------------+-------------+----------------
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+(2 rows)
+
+-- What a subscription to both ipub_root and ipub_part1 should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+------------+---------------
+ ipub_root | sch1.iroot
+ ipub_root | sch1.ipart1
+ ipub_root | sch1.ipart2
+ ipub_root | sch1.ipart1_a
+ ipub_part1 | sch1.ipart1
+(5 rows)
+
+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
+ published | synced
+---------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a | sch1.ipart1_a
+(5 rows)
+
+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
+ pubname | published | synced
+------------+---------------+----------------
+ ipub_root | sch1.iroot | sch1.iroot
+ ipub_root | sch1.ipart1 | sch1.ipart1
+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
+ ipub_root | sch1.ipart2 | sch1.ipart2
+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+(7 rows)
+
+-- What a subscription to both ipub_part1 and ipub_other should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+------------+-------------
+ ipub_part1 | sch1.ipart1
+ ipub_other | sch1.iroot
+(2 rows)
+
+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
+ published | synced
+-------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.iroot | sch1.ipart2
+ sch1.iroot | sch1.ipart1_a
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
+ pubname | published | synced
+------------+-------------+----------------
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+ ipub_other | sch1.iroot | sch1.iroot
+ ipub_other | sch1.iroot | sch1.ipart2
+ ipub_other | sch1.iroot | sch1.ipart1_a
+(5 rows)
+
+-- What a subscription to all three should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+------------+-------------
+ ipub_root | sch1.iroot
+ ipub_root | sch1.ipart1
+ ipub_part1 | sch1.ipart1
+ ipub_other | sch1.iroot
+(4 rows)
+
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+ published | synced
+-------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.iroot | sch1.ipart2
+ sch1.iroot | sch1.ipart1_a
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+ pubname | published | synced
+------------+-------------+----------------
+ ipub_root | sch1.iroot | sch1.iroot
+ ipub_root | sch1.iroot | sch1.ipart2
+ ipub_root | sch1.iroot | sch1.ipart1_a
+ ipub_root | sch1.ipart1 | sch1.ipart1
+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+ ipub_other | sch1.iroot | sch1.iroot
+ ipub_other | sch1.iroot | sch1.ipart2
+ ipub_other | sch1.iroot | sch1.ipart1_a
+(10 rows)
+
+-- "Detaching" partitions should change the subscriptions
+ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+------------+-------------
+ ipub_root | sch1.iroot
+ ipub_root | sch1.ipart1
+ ipub_root | sch1.ipart2
+ ipub_part1 | sch1.ipart1
+ ipub_other | sch1.iroot
+ ipub_other | sch1.ipart2
+(6 rows)
+
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+ published | synced
+-------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.iroot | sch1.ipart1_a
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a1
+ sch1.ipart2 | sch1.ipart2
+(5 rows)
+
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+ pubname | published | synced
+------------+-------------+----------------
+ ipub_root | sch1.iroot | sch1.iroot
+ ipub_root | sch1.iroot | sch1.ipart1_a
+ ipub_root | sch1.ipart1 | sch1.ipart1
+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
+ ipub_root | sch1.ipart2 | sch1.ipart2
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
+ ipub_other | sch1.iroot | sch1.iroot
+ ipub_other | sch1.iroot | sch1.ipart1_a
+ ipub_other | sch1.ipart2 | sch1.ipart2
+(10 rows)
+
+ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
+ relid
+----------------
+ sch1.ipart1
+ sch1.ipart1_a1
+(2 rows)
+
+SELECT * FROM published_sync('ipub_part1');
+ published | synced
+----------------+----------------
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(2 rows)
+
+SELECT * FROM published_stream('ipub_part1');
+ pubname | published | synced
+------------+----------------+----------------
+ ipub_part1 | sch1.ipart1 | sch1.ipart1
+ ipub_part1 | sch1.ipart1_a1 | sch1.ipart1_a1
+(2 rows)
+
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1_a1;
+DROP TABLE sch1.ipart1_a;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
DROP FUNCTION published_stream;
+DROP FUNCTION published_sync;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
DROP ROLE regress_publication_user_dummy;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 6bf9554d48..d1ea777ef9 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -6,6 +6,18 @@ CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_sync (VARIADIC pubnames text[])
+ RETURNS TABLE (published regclass, synced regclass) AS $$
+ -- Show each published table alongside the tables to be copied into it during
+ -- initial sync.
+ SELECT t.published, synced
+ FROM (SELECT DISTINCT relid::regclass AS published
+ FROM pg_get_publication_tables(VARIADIC pubnames)) t
+ JOIN LATERAL pg_get_publication_rels_to_sync(t.published, VARIADIC pubnames) synced
+ ON true
+ ORDER BY 1, 2;
+$$ LANGUAGE sql;
+
CREATE FUNCTION published_stream (VARIADIC pubnames text[])
RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
-- For each publication, show each published root alongside the tables which
@@ -1076,6 +1088,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_sync('pub');
SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
@@ -1091,6 +1104,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_sync('pub');
SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
@@ -1105,12 +1119,138 @@ ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
+-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+
+-- should do nothing at all
+ALTER TABLE sch1.iroot SET (publish_via_parent = false);
+ALTER TABLE sch1.iroot RESET (publish_via_parent);
+
+-- marking roots between unrelated tables is not allowed
+ALTER TABLE sch1.ipart1 SET (publish_via_parent);
+CREATE TABLE fail (a int) WITH (publish_via_parent);
+
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
+ SET (publish_via_parent);
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
+ SET (publish_via_parent);
+
+-- once publish_via_parent is set, inheritance cannot be changed
+ALTER TABLE sch1.ipart2 SET (publish_via_parent);
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
+
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should fail
+RESET SESSION AUTHORIZATION;
+
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should also fail
+CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+RESET SESSION AUTHORIZATION;
+
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+
+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
+ WITH (publish_via_partition_root);
+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
+ ONLY sch1.ipart2
+ WITH (publish_via_partition_root);
+
+-- At this point, the published trees look like this:
+--
+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
+-- ------------------ ----------------------- -----------------------
+-- iroot iroot
+-- +- ipart1 ipart1 |
+-- | +- ipart1_a | +--- ipart1_a
+-- | +- ipart1_a1 +--- ipart1_a1 |
+-- +- ipart2 +- ipart2
+
+-- What a subscription to only ipub_root should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
+SELECT * FROM published_sync('ipub_root');
+SELECT * FROM published_stream('ipub_root');
+
+-- What a subscription to only ipub_part1 should see
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
+SELECT * FROM published_sync('ipub_part1');
+SELECT * FROM published_stream('ipub_part1');
+
+-- What a subscription to both ipub_root and ipub_part1 should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
+
+-- What a subscription to both ipub_part1 and ipub_other should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
+
+-- What a subscription to all three should see
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+
+-- "Detaching" partitions should change the subscriptions
+ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
+
+ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
+SELECT * FROM published_sync('ipub_part1');
+SELECT * FROM published_stream('ipub_part1');
+
+DROP PUBLICATION ipub_other;
+DROP PUBLICATION ipub_part1;
+DROP PUBLICATION ipub_root;
+
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1_a1;
+DROP TABLE sch1.ipart1_a;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
DROP FUNCTION published_stream;
+DROP FUNCTION published_sync;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 275fb3b525..09e3f2d028 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -388,6 +388,59 @@ $node_subscriber1->append_conf('postgresql.conf',
"log_min_messages = warning");
$node_subscriber1->reload;
+# Make sure standard inheritance setups aren't broken by the new
+# publish_via_parent handling.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (LIKE itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (LIKE itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_2 VALUES (2, 'itab1')");
+
+# Regression: Create a publication for an unrelated table and set
+# publish_via_partition_root. This should have no effect at all, since itab1
+# isn't supposed to be using this publication...
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION dummy_pub FOR TABLE tab1 WITH (publish_via_partition_root = true)");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 ADD PUBLICATION dummy_pub");
+$node_subscriber2->wait_for_subscription_sync;
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1");
+is($result, qq(0|itab1), 'initial data synced for itab1 on subscriber 2');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_1");
+is($result, qq(1|itab1), 'initial data synced for itab1_1 on subscriber 2');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_2");
+is($result, qq(2|itab1), 'initial data synced for itab1_2 on subscriber 2');
+
+$node_publisher->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+$node_subscriber2->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 DROP PUBLICATION dummy_pub");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION dummy_pub");
+
# Tests for replication using root table identity and schema
# publisher
@@ -886,4 +939,263 @@ $result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab5_1 ORDER BY 1");
is($result, qq(4||1), 'updates of tab5 replicated correctly');
+# Test that replication works for older inheritance/trigger setups as well.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab1_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab1_1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO itab1_2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab1_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab1_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (CHECK (a = 1)) INHERITS (itab2)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab2_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab2_1 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab2_trigger
+ BEFORE INSERT ON itab2
+ FOR EACH ROW EXECUTE FUNCTION itab2_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab2_1 SET (publish_via_parent = true)");
+
+# itab2_1 should be published using its own identity here, since its parent is
+# not included. itab1_1 should be published via its parent, itab1, without
+# duplicating the rows.
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub_viaroot ADD TABLE itab1, itab1_1, itab2_1");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (2, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (0, 'itab2')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2')");
+
+# Subscriber 1 only subscribes to some of the partitions, and does not set up
+# partition triggers, to check for the correct routing.
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (a int, b text)");
+
+# Subscriber 2 has different partition names for itab1, and it doesn't partition
+# itab2 at all.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_subscriber2->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO public.itab1_part1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO public.itab1_part2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_subscriber2->safe_psql('postgres', "
+ CREATE TRIGGER itab_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab_trigger();");
+$node_subscriber2->safe_psql('postgres', "
+ ALTER TABLE itab1 ENABLE ALWAYS TRIGGER itab_trigger;");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot REFRESH PUBLICATION");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 REFRESH PUBLICATION");
+
+$node_subscriber1->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
+
+# check that data is synced correctly
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 1');
+
+# all of the data should have been routed to itab1 directly (there are no
+# triggers on subscriber 1 to move it elsewhere)
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data correctly routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2), 'initial data synced for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1), 'initial data correctly routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2), 'initial data synced for itab2 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1-new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2-new')");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+$node_publisher->wait_for_catchup('sub2');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2
+1|itab2-new), 'new data routed for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_part1 ORDER BY 1, 2");
+is($result, qq(1|itab1
+1|itab1-new), 'new data moved to itab1_part1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2
+1|itab2-new), 'new data routed for itab2 on subscriber 2');
+
+# Finally, check the effect of mixed publish_via_partition_root settings on
+# logical roots.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3_1 () INHERITS (itab3)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab3_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab3_1_1 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3 VALUES (1, 'itab3')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3_1 VALUES (2, 'itab3_1')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3_1_1 VALUES (3, 'itab3_1_1')");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION ipub3 FOR TABLE itab3 *");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION ipub3_1 FOR TABLE itab3_1 * WITH (publish_via_partition_root = true)");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3_1 () INHERITS (itab3)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION mixed CONNECTION '$publisher_connstr' PUBLICATION ipub3, ipub3_1");
+$node_subscriber2->wait_for_subscription_sync;
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3");
+is($result, qq(1|itab3), 'initial data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1), 'initial data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
+is($result, qq(), 'initial data routed for itab3_1_1 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3 VALUES (4, 'itab3_new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1 VALUES (4, 'itab3_1_new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1_1 VALUES (4, 'itab3_1_1_new')");
+
+$node_publisher->wait_for_catchup('mixed');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3 ORDER BY 1, 2");
+is($result, qq(1|itab3
+4|itab3_new), 'new data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1
+4|itab3_1_1_new
+4|itab3_1_new), 'new data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
+is($result, qq(), 'new data routed for itab3_1_1 on subscriber 2');
+
done_testing();
--
2.25.1
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-04-04 03:53 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
2023-04-04 15:14 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-06-06 15:50 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-06-29 23:46 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
@ 2023-07-19 11:54 ` Aleksander Alekseev <[email protected]>
2023-08-31 18:16 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Aleksander Alekseev @ 2023-07-19 11:54 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: Jacob Champion <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>
Hi,
> v3 also fixes a nasty uninitialized stack variable, along with a bad
> collation assumption I made.
I decided to take a closer look at 0001.
Since pg_get_relation_publishing_info() is exposed to the users I
think it should be described in a bit more detail than:
```
+ descr => 'get information on how a relation will be published via a
list of publications',
```
This description in \df+ output doesn't seem to be particularly
useful. Also the function should be documented. In order to accomplish
all this it could make sense to reconsider the signature of the
function and/or split it into several separate functions.
The volatility is declared as STABLE. This is probably correct. At
least at first glance I don't see any calls of VOLATILE functions and
off the top of my head can't give an example when it will not behave
as STABLE. This being said, a second opinion would be appreciated.
process_relation_publications() misses a brief comment before the
declaration. What are the arguments, what is the return value, are
there any pre/postconditions (locks, memory), etc.
Otherwise 0001 is in a decent shape, it passes make
installcheck-world, etc. I would suggest focusing on delivering this
part, assuming there will be no push-back to the refactorings and
slight test improvements. If 0002 could be further decomposed into
separate iterative improvements this could be helpful.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: RFC: logical publication via inheritance root?
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-20 17:53 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-02-28 22:47 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-04-04 03:53 ` Re: RFC: logical publication via inheritance root? Peter Smith <[email protected]>
2023-04-04 15:14 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-06-06 15:50 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-06-29 23:46 ` Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-07-19 11:54 ` Re: RFC: logical publication via inheritance root? Aleksander Alekseev <[email protected]>
@ 2023-08-31 18:16 ` Jacob Champion <[email protected]>
0 siblings, 0 replies; 18+ messages in thread
From: Jacob Champion @ 2023-08-31 18:16 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; PostgreSQL Hackers <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>
On 7/19/23 04:54, Aleksander Alekseev wrote:
> I decided to take a closer look at 0001.
Hi Aleks! I saw you put this back into Needs Review; thanks.
This thread has been pretty quiet from me, because we've run into
difficulties on the subscriber end. Our original optimistic assumption
was that we just needed to funnel all the leaf tables through the root
on the publisher, and then a sufficiently complex replica trigger on
the subscriber would be able to correctly route the data through the
root into new leaves.
That has not panned out for a number of reasons. Probably the easiest
one to describe is that replica identity handling breaks: if we move
the incoming tuples to different tables, and an UPDATE or DELETE comes
in later for those rows, the replication logic checks the root table
(bypassing our existing routing logic) and sees that they don't exist.
We never get the chance to handle routing the way that partitions do
[1]. Given that, I think I need to pivot and focus on the subscriber
side first. That might(?) be a smaller effort anyway, and if we can't
make headway there then publisher-side support probably doesn't make
sense at all.
So I'll pause this CF entry for now. This would also be a good time to
ask the crowd: are there alternative approaches to solve the OP that I
may be missing?
Thanks!
--Jacob
[1] https://git.postgresql.org/cgit/postgresql.git/tree/src/backend/replication/logical/worker.c?h=8bf7d...
P.S. I've attached a v4, which fixes the semantics problem I mentioned
upthread, so it doesn't get lost in the shuffle.
1: 7e68f96af6 = 1: 173e74279c pgoutput: refactor publication cache construction
2: a48563919e ! 2: dc2425ceaa WIP: introduce publish_via_parent for logical replication
@@ Commit message
roots.
Known bugs/TODOs:
- - If two publications publish the same leaves of a table hierarchy, but
- have different roots, the shared leaf tables will be incorrectly
- duplicated when subscribing to both publications simultaneously.
- I haven't given any thought to interactions with row filters, or to
column lists.
- I'm not sure that I'm taking all the necessary locks yet, and those I
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
+ {
+ Oid rootid = PG_GETARG_OID(0);
+ MemoryContext oldcontext;
-+ ArrayType *arr;
-+ Datum *elems;
-+ int nelems,
-+ i;
++ List *publications = NIL;
++ List *inheritors;
++ List *candidates = NIL;
++ ListCell *lc;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
++ /* Construct the list of publications to consider. */
++ {
++ ArrayType *arr;
++ Datum *elems;
++ int nelems,
++ i;
++
++ /*
++ * Deconstruct the parameter into elements where each element is a
++ * publication name.
++ */
++ arr = PG_GETARG_ARRAYTYPE_P(1);
++ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT,
++ &elems, NULL, &nelems);
++
++ for (i = 0; i < nelems; i++)
++ {
++ char *pubname = TextDatumGetCString(elems[i]);
++ Publication *pub = GetPublicationByName(pubname, false);
++
++ publications = lappend(publications, pub);
++ }
++ }
++
++ /* TODO: do the tables in this list need to be locked? */
++ inheritors = find_all_logical_inheritors(rootid);
++
+ /*
-+ * Deconstruct the parameter into elements where each element is a
-+ * publication name.
++ * For a logical descendant to be in the list to sync, both it and the
++ * root table need to be part of a pubviaroot publication. (But they
++ * don't have to be part of the *same* pubviaroot publication. This is
++ * somewhat counterintuitive, but it prevents a descendant from being
++ * double-published through two separate roots.)
+ */
-+ arr = PG_GETARG_ARRAYTYPE_P(1);
-+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT,
-+ &elems, NULL, &nelems);
-+
-+ /* Get Oids of tables from each publication. */
-+ for (i = 0; i < nelems; i++)
++ foreach(lc, publications)
+ {
-+ char *pubname = TextDatumGetCString(elems[i]);
-+ Publication *publication = GetPublicationByName(pubname, false);
-+ List *pub_tables = NIL;
++ Publication *publication = lfirst(lc);
++ List *relids;
++ List *schemarelids;
++ List *published;
++ ListCell *cell;
+
-+ /* TODO: do the tables in this list need to be locked? */
-+ if (publication->pubviaroot)
-+ pub_tables = find_all_logical_inheritors(rootid);
-+ else
-+ pub_tables = list_make1_oid(rootid);
++ if (!publication->pubviaroot)
++ continue;
+
-+ if (!publication->alltables)
++ if (publication->alltables)
+ {
-+ List *relids;
-+ List *schemarelids;
-+ List *published;
-+ ListCell *cell;
-+
-+ relids = GetPublicationRelations(publication->oid,
-+ publication->pubviaroot ?
-+ PUBLICATION_PART_ROOT :
-+ PUBLICATION_PART_ALL);
-+ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
-+ publication->pubviaroot ?
-+ PUBLICATION_PART_ROOT :
-+ PUBLICATION_PART_LEAF);
-+ published = list_concat_unique_oid(relids, schemarelids);
-+
-+ /*
-+ * First we have to check to make sure the root table is
-+ * actually part of the publication; if not, none of its
-+ * descendants' contents belong to it.
-+ */
-+ if (!list_member_oid(published, rootid))
-+ continue;
-+
-+ /*
-+ * Now filter out any descendants that aren't part of this
-+ * particular publication.
-+ */
-+ foreach(cell, pub_tables)
-+ {
-+ Oid current = lfirst_oid(cell);
-+
-+ if (!list_member_oid(published, current))
-+ pub_tables = foreach_delete_current(pub_tables, cell);
-+ }
++ /* Easy case: every inheritor is published through the root. */
++ candidates = inheritors;
++ break;
+ }
+
-+ tables = list_concat_unique_oid(tables, pub_tables);
++ /*
++ * Figure out which of the inheritors is included in this
++ * publication. They get added to our candidates list.
++ */
++ relids = GetPublicationRelations(publication->oid,
++ publication->pubviaroot ?
++ PUBLICATION_PART_ROOT :
++ PUBLICATION_PART_ALL);
++ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
++ publication->pubviaroot ?
++ PUBLICATION_PART_ROOT :
++ PUBLICATION_PART_LEAF);
++ published = list_concat_unique_oid(relids, schemarelids);
++
++ foreach(cell, inheritors)
++ {
++ Oid current = lfirst_oid(cell);
++
++ if (list_member_oid(published, current))
++ candidates = list_append_unique_oid(candidates, current);
++ }
+ }
+
++ /*
++ * Finally, if we didn't find the root table itself in a pubviaroot
++ * publication, none of the logical descendants will be published
++ * through it.
++ *
++ * XXX: this assumes that the root table was part of at least one of the
++ * publications in the list. It doesn't make any sense to call this
++ * function otherwise, but should we ERROR out if that assumption is
++ * violated?
++ */
++ if (list_member_oid(candidates, rootid))
++ tables = candidates;
++ else
++ tables = list_make1_oid(rootid);
++
+ funcctx->user_fctx = (void *) tables;
+
+ MemoryContextSwitchTo(oldcontext);
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
process_relation_publications(Oid relid, const List *publications,
PublicationActions *pubactions,
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, const List *publications,
+ ListCell *lc;
+ int publish_ancestor_level = 0;
bool am_partition = get_rel_relispartition(relid);
++ bool am_pubviaroot = false;
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
--
-- *publish_as_relid = relid;
+ Oid publish_candidate = relid;
++
++ /*
++ * For publish_via_parent handling later, it's useful to know whether the
++ * current table is explicitly part of a publish_via_partition_root
++ * publication up front.
++ */
++ foreach(lc, publications)
++ {
++ Publication *pub = lfirst(lc);
+
+- *publish_as_relid = relid;
++ if (!pub->pubviaroot)
++ continue;
++
++ if (pub->alltables ||
++ list_member_oid(pubids, pub->oid) ||
++ list_member_oid(schemaPubids, pub->oid))
++ {
++ am_pubviaroot = true;
++ break;
++ }
++ }
foreach(lc, publications)
{
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
+ }
}
}
-
- if (!publish)
+-
+- if (!publish)
++ else
{
bool ancestor_published = false;
+ Oid ancestor;
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
--
-- ancestor = GetTopMostAncestorInPublication(pub->oid,
-- ancestors,
-- &level);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
+- ancestor = GetTopMostAncestorInPublication(pub->oid,
+- ancestors,
+- &level);
+-
- if (ancestor != InvalidOid)
+ if (ancestor != InvalidOid)
+ {
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
if (list_member_oid(pubids, pub->oid) ||
list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
-+ (am_partition && ancestor_published))
++ (ancestor_published && am_partition) ||
++ (ancestor_published && am_pubviaroot && pub->pubviaroot))
publish = true;
}
@@ src/backend/commands/publicationcmds.c: TransformPubWhereClauses(List *tables, c
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
-+ * TODO: decide how this interacts with publish_via_parent
++ * TODO: decide how this interacts with publish_via_parent. Probably not
++ * in the same way, since logical roots can contain data of their own.
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ src/backend/commands/publicationcmds.c: CheckPubRelationColumnList(char *pubname
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
-+ * TODO: decide if this interacts with publish_via_parent
++ * TODO: decide if this interacts with publish_via_parent. Probably not
++ * (see above).
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
-+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
-+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
++CREATE PUBLICATION ipub_all FOR TABLE sch1.iroot;
++CREATE PUBLICATION ipub_one FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
-+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
-+ ONLY sch1.ipart2
++CREATE PUBLICATION ipub_two FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
-+-- At this point, the published trees look like this:
-+--
-+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
-+-- ------------------ ----------------------- -----------------------
-+-- iroot iroot
-+-- +- ipart1 ipart1 |
-+-- | +- ipart1_a | +--- ipart1_a
-+-- | +- ipart1_a1 +--- ipart1_a1 |
-+-- +- ipart2 +- ipart2
-+-- What a subscription to only ipub_root should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1
++-- | +- ipart1_a - ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++-- A subscription to only ipub_all should see every individual table.
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_all');
+ relid
+----------------
+ sch1.iroot
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_sync('ipub_root');
++SELECT * FROM published_sync('ipub_all');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_root');
-+ pubname | published | synced
-+-----------+----------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart2 | sch1.ipart2
-+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
-+ ipub_root | sch1.ipart1_a1 | sch1.ipart1_a1
++SELECT * FROM published_stream('ipub_all');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+-- What a subscription to only ipub_part1 should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+ relid
-+-------------
-+ sch1.ipart1
-+(1 row)
++-- A subscription to both ipub_all and ipub_one shouldn't change the initial
++-- sync, since there is no alternative root being published for ipart1_a. It
++-- will be duplicated in the stream list, since TODO
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart1
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1_a
++(6 rows)
++
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a | sch1.ipart1_a
++ sch1.ipart1_a1 | sch1.ipart1_a1
++(5 rows)
++
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.ipart1_a | sch1.ipart1_a
++(6 rows)
+
-+SELECT * FROM published_sync('ipub_part1');
-+ published | synced
-+-------------+----------------
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
-+(2 rows)
-+
-+SELECT * FROM published_stream('ipub_part1');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+(2 rows)
-+
-+-- What a subscription to both ipub_root and ipub_part1 should see
++ALTER PUBLICATION ipub_one ADD TABLE ONLY sch1.ipart1;
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1 - ipart1
++-- | +- ipart1_a +- ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++-- Adding ipart1_a's parent table to ipub_one results in ipart1_a1 being
++-- "stolen" from the ipub_all stream, to prevent its data from being duplicated.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+---------------
-+ ipub_root | sch1.iroot
-+ ipub_root | sch1.ipart1
-+ ipub_root | sch1.ipart2
-+ ipub_root | sch1.ipart1_a
-+ ipub_part1 | sch1.ipart1
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart1
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1
+(5 rows)
+
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
-+ published | synced
-+---------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
-+ sch1.ipart2 | sch1.ipart2
-+ sch1.ipart1_a | sch1.ipart1_a
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1 | sch1.ipart1_a
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
-+ pubname | published | synced
-+------------+---------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_root | sch1.ipart2 | sch1.ipart2
-+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart1 | sch1.ipart1_a
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.ipart1 | sch1.ipart1
++ ipub_one | sch1.ipart1 | sch1.ipart1_a
+(7 rows)
+
-+-- What a subscription to both ipub_part1 and ipub_other should see
++-- Including ipub_two in the subscription list should change nothing about the
++-- sync. ipub_two will gain an entry for ipart1_a in the streaming list.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+-------------
-+ ipub_part1 | sch1.ipart1
-+ ipub_other | sch1.iroot
-+(2 rows)
-+
-+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
-+ published | synced
-+-------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.iroot | sch1.ipart2
-+ sch1.iroot | sch1.ipart1_a
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart1
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_other | sch1.iroot | sch1.iroot
-+ ipub_other | sch1.iroot | sch1.ipart2
-+ ipub_other | sch1.iroot | sch1.ipart1_a
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1 | sch1.ipart1_a
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+-- What a subscription to all three should see
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart1 | sch1.ipart1_a
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.ipart1 | sch1.ipart1
++ ipub_one | sch1.ipart1 | sch1.ipart1_a
++ ipub_two | sch1.ipart1 | sch1.ipart1_a
++(8 rows)
++
++ALTER PUBLICATION ipub_two ADD TABLE ONLY sch1.iroot;
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot iroot
++-- +- ipart1 - ipart1 |
++-- | +- ipart1_a +- ipart1_a +---- ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++-- Adding iroot to ipub_two ends up stealing both ipart1 and ipart1_a from the
++-- other publications, again so that no data is double-published.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+-------------
-+ ipub_root | sch1.iroot
-+ ipub_root | sch1.ipart1
-+ ipub_part1 | sch1.ipart1
-+ ipub_other | sch1.iroot
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a1
++ ipub_two | sch1.iroot
+(4 rows)
+
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+ published | synced
-+-------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.iroot | sch1.ipart2
-+ sch1.iroot | sch1.ipart1_a
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.iroot | sch1.ipart1
++ sch1.iroot | sch1.ipart1_a
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.iroot | sch1.ipart2
-+ ipub_root | sch1.iroot | sch1.ipart1_a
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_other | sch1.iroot | sch1.iroot
-+ ipub_other | sch1.iroot | sch1.ipart2
-+ ipub_other | sch1.iroot | sch1.ipart1_a
-+(10 rows)
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.iroot | sch1.ipart1
++ ipub_all | sch1.iroot | sch1.ipart1_a
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.iroot | sch1.ipart1
++ ipub_one | sch1.iroot | sch1.ipart1_a
++ ipub_two | sch1.iroot | sch1.iroot
++ ipub_two | sch1.iroot | sch1.ipart1_a
++(9 rows)
+
+-- "Detaching" partitions should change the subscriptions
-+ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++ALTER TABLE sch1.ipart1_a SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+-------------
-+ ipub_root | sch1.iroot
-+ ipub_root | sch1.ipart1
-+ ipub_root | sch1.ipart2
-+ ipub_part1 | sch1.ipart1
-+ ipub_other | sch1.iroot
-+ ipub_other | sch1.ipart2
-+(6 rows)
-+
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+ published | synced
-+-------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.iroot | sch1.ipart1_a
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
-+ sch1.ipart2 | sch1.ipart2
-+(5 rows)
-+
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.iroot | sch1.ipart1_a
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_root | sch1.ipart2 | sch1.ipart2
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_other | sch1.iroot | sch1.iroot
-+ ipub_other | sch1.iroot | sch1.ipart1_a
-+ ipub_other | sch1.ipart2 | sch1.ipart2
-+(10 rows)
-+
-+ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+ relid
-+----------------
-+ sch1.ipart1
-+ sch1.ipart1_a1
-+(2 rows)
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1_a
++ ipub_two | sch1.iroot
++ ipub_two | sch1.ipart1_a
++(7 rows)
+
-+SELECT * FROM published_sync('ipub_part1');
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+ published | synced
+----------------+----------------
-+ sch1.ipart1 | sch1.ipart1
++ sch1.iroot | sch1.iroot
++ sch1.iroot | sch1.ipart1
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a | sch1.ipart1_a
+ sch1.ipart1_a1 | sch1.ipart1_a1
-+(2 rows)
-+
-+SELECT * FROM published_stream('ipub_part1');
-+ pubname | published | synced
-+------------+----------------+----------------
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1_a1 | sch1.ipart1_a1
-+(2 rows)
-+
-+DROP PUBLICATION ipub_other;
-+DROP PUBLICATION ipub_part1;
-+DROP PUBLICATION ipub_root;
++(5 rows)
++
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.iroot | sch1.ipart1
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.iroot | sch1.ipart1
++ ipub_one | sch1.ipart1_a | sch1.ipart1_a
++ ipub_two | sch1.iroot | sch1.iroot
++ ipub_two | sch1.ipart1_a | sch1.ipart1_a
++(9 rows)
++
++ALTER TABLE sch1.ipart1 RESET (publish_via_parent);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_one', 'ipub_two') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++ pubname | relid
++----------+---------------
++ ipub_one | sch1.ipart1
++ ipub_one | sch1.ipart1_a
++ ipub_two | sch1.iroot
++ ipub_two | sch1.ipart1_a
++(4 rows)
++
++SELECT * FROM published_sync('ipub_one', 'ipub_two');
++ published | synced
++---------------+---------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1_a | sch1.ipart1_a
++(3 rows)
++
++SELECT * FROM published_stream('ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+---------------+---------------
++ ipub_one | sch1.ipart1 | sch1.ipart1
++ ipub_one | sch1.ipart1_a | sch1.ipart1_a
++ ipub_two | sch1.iroot | sch1.iroot
++ ipub_two | sch1.ipart1_a | sch1.ipart1_a
++(4 rows)
++
++DROP PUBLICATION ipub_all;
++DROP PUBLICATION ipub_one;
++DROP PUBLICATION ipub_two;
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
@@ src/test/regress/sql/publication.sql: ALTER TABLE sch1.tbl1 ATTACH PARTITION sch
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+
-+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
-+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
++CREATE PUBLICATION ipub_all FOR TABLE sch1.iroot;
++CREATE PUBLICATION ipub_one FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
-+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
-+ ONLY sch1.ipart2
++CREATE PUBLICATION ipub_two FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
+
-+-- At this point, the published trees look like this:
-+--
-+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
-+-- ------------------ ----------------------- -----------------------
-+-- iroot iroot
-+-- +- ipart1 ipart1 |
-+-- | +- ipart1_a | +--- ipart1_a
-+-- | +- ipart1_a1 +--- ipart1_a1 |
-+-- +- ipart2 +- ipart2
-+
-+-- What a subscription to only ipub_root should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
-+SELECT * FROM published_sync('ipub_root');
-+SELECT * FROM published_stream('ipub_root');
-+
-+-- What a subscription to only ipub_part1 should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+SELECT * FROM published_sync('ipub_part1');
-+SELECT * FROM published_stream('ipub_part1');
-+
-+-- What a subscription to both ipub_root and ipub_part1 should see
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1
++-- | +- ipart1_a - ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++
++-- A subscription to only ipub_all should see every individual table.
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_all');
++SELECT * FROM published_sync('ipub_all');
++SELECT * FROM published_stream('ipub_all');
++
++-- A subscription to both ipub_all and ipub_one shouldn't change the initial
++-- sync, since there is no alternative root being published for ipart1_a. It
++-- will be duplicated in the stream list, since TODO
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
++
++ALTER PUBLICATION ipub_one ADD TABLE ONLY sch1.ipart1;
++
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1 - ipart1
++-- | +- ipart1_a +- ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
+
-+-- What a subscription to both ipub_part1 and ipub_other should see
++-- Adding ipart1_a's parent table to ipub_one results in ipart1_a1 being
++-- "stolen" from the ipub_all stream, to prevent its data from being duplicated.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
-+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
+
-+-- What a subscription to all three should see
++-- Including ipub_two in the subscription list should change nothing about the
++-- sync. ipub_two will gain an entry for ipart1_a in the streaming list.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++
++ALTER PUBLICATION ipub_two ADD TABLE ONLY sch1.iroot;
++
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot iroot
++-- +- ipart1 - ipart1 |
++-- | +- ipart1_a +- ipart1_a +---- ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++
++-- Adding iroot to ipub_two ends up stealing both ipart1 and ipart1_a from the
++-- other publications, again so that no data is double-published.
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+
+-- "Detaching" partitions should change the subscriptions
-+ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++ALTER TABLE sch1.ipart1_a SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+
-+ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+SELECT * FROM published_sync('ipub_part1');
-+SELECT * FROM published_stream('ipub_part1');
++ALTER TABLE sch1.ipart1 RESET (publish_via_parent);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_one', 'ipub_two') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++SELECT * FROM published_sync('ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_one', 'ipub_two');
+
-+DROP PUBLICATION ipub_other;
-+DROP PUBLICATION ipub_part1;
-+DROP PUBLICATION ipub_root;
++DROP PUBLICATION ipub_all;
++DROP PUBLICATION ipub_one;
++DROP PUBLICATION ipub_two;
+
RESET client_min_messages;
DROP PUBLICATION pub;
Attachments:
[text/plain] since-v3.diff.txt (36.8K, ../../CAAWbhmjcnoV7Xu6LHr_hxqWmVtehv404bvDye+QZcUDSg8NSKw@mail.gmail.com/2-since-v3.diff.txt)
download | inline:
1: 7e68f96af6 = 1: 173e74279c pgoutput: refactor publication cache construction
2: a48563919e ! 2: dc2425ceaa WIP: introduce publish_via_parent for logical replication
@@ Commit message
roots.
Known bugs/TODOs:
- - If two publications publish the same leaves of a table hierarchy, but
- have different roots, the shared leaf tables will be incorrectly
- duplicated when subscribing to both publications simultaneously.
- I haven't given any thought to interactions with row filters, or to
column lists.
- I'm not sure that I'm taking all the necessary locks yet, and those I
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
+ {
+ Oid rootid = PG_GETARG_OID(0);
+ MemoryContext oldcontext;
-+ ArrayType *arr;
-+ Datum *elems;
-+ int nelems,
-+ i;
++ List *publications = NIL;
++ List *inheritors;
++ List *candidates = NIL;
++ ListCell *lc;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
++ /* Construct the list of publications to consider. */
++ {
++ ArrayType *arr;
++ Datum *elems;
++ int nelems,
++ i;
++
++ /*
++ * Deconstruct the parameter into elements where each element is a
++ * publication name.
++ */
++ arr = PG_GETARG_ARRAYTYPE_P(1);
++ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT,
++ &elems, NULL, &nelems);
++
++ for (i = 0; i < nelems; i++)
++ {
++ char *pubname = TextDatumGetCString(elems[i]);
++ Publication *pub = GetPublicationByName(pubname, false);
++
++ publications = lappend(publications, pub);
++ }
++ }
++
++ /* TODO: do the tables in this list need to be locked? */
++ inheritors = find_all_logical_inheritors(rootid);
++
+ /*
-+ * Deconstruct the parameter into elements where each element is a
-+ * publication name.
++ * For a logical descendant to be in the list to sync, both it and the
++ * root table need to be part of a pubviaroot publication. (But they
++ * don't have to be part of the *same* pubviaroot publication. This is
++ * somewhat counterintuitive, but it prevents a descendant from being
++ * double-published through two separate roots.)
+ */
-+ arr = PG_GETARG_ARRAYTYPE_P(1);
-+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT,
-+ &elems, NULL, &nelems);
-+
-+ /* Get Oids of tables from each publication. */
-+ for (i = 0; i < nelems; i++)
++ foreach(lc, publications)
+ {
-+ char *pubname = TextDatumGetCString(elems[i]);
-+ Publication *publication = GetPublicationByName(pubname, false);
-+ List *pub_tables = NIL;
++ Publication *publication = lfirst(lc);
++ List *relids;
++ List *schemarelids;
++ List *published;
++ ListCell *cell;
+
-+ /* TODO: do the tables in this list need to be locked? */
-+ if (publication->pubviaroot)
-+ pub_tables = find_all_logical_inheritors(rootid);
-+ else
-+ pub_tables = list_make1_oid(rootid);
++ if (!publication->pubviaroot)
++ continue;
+
-+ if (!publication->alltables)
++ if (publication->alltables)
+ {
-+ List *relids;
-+ List *schemarelids;
-+ List *published;
-+ ListCell *cell;
-+
-+ relids = GetPublicationRelations(publication->oid,
-+ publication->pubviaroot ?
-+ PUBLICATION_PART_ROOT :
-+ PUBLICATION_PART_ALL);
-+ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
-+ publication->pubviaroot ?
-+ PUBLICATION_PART_ROOT :
-+ PUBLICATION_PART_LEAF);
-+ published = list_concat_unique_oid(relids, schemarelids);
-+
-+ /*
-+ * First we have to check to make sure the root table is
-+ * actually part of the publication; if not, none of its
-+ * descendants' contents belong to it.
-+ */
-+ if (!list_member_oid(published, rootid))
-+ continue;
-+
-+ /*
-+ * Now filter out any descendants that aren't part of this
-+ * particular publication.
-+ */
-+ foreach(cell, pub_tables)
-+ {
-+ Oid current = lfirst_oid(cell);
-+
-+ if (!list_member_oid(published, current))
-+ pub_tables = foreach_delete_current(pub_tables, cell);
-+ }
++ /* Easy case: every inheritor is published through the root. */
++ candidates = inheritors;
++ break;
+ }
+
-+ tables = list_concat_unique_oid(tables, pub_tables);
++ /*
++ * Figure out which of the inheritors is included in this
++ * publication. They get added to our candidates list.
++ */
++ relids = GetPublicationRelations(publication->oid,
++ publication->pubviaroot ?
++ PUBLICATION_PART_ROOT :
++ PUBLICATION_PART_ALL);
++ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
++ publication->pubviaroot ?
++ PUBLICATION_PART_ROOT :
++ PUBLICATION_PART_LEAF);
++ published = list_concat_unique_oid(relids, schemarelids);
++
++ foreach(cell, inheritors)
++ {
++ Oid current = lfirst_oid(cell);
++
++ if (list_member_oid(published, current))
++ candidates = list_append_unique_oid(candidates, current);
++ }
+ }
+
++ /*
++ * Finally, if we didn't find the root table itself in a pubviaroot
++ * publication, none of the logical descendants will be published
++ * through it.
++ *
++ * XXX: this assumes that the root table was part of at least one of the
++ * publications in the list. It doesn't make any sense to call this
++ * function otherwise, but should we ERROR out if that assumption is
++ * violated?
++ */
++ if (list_member_oid(candidates, rootid))
++ tables = candidates;
++ else
++ tables = list_make1_oid(rootid);
++
+ funcctx->user_fctx = (void *) tables;
+
+ MemoryContextSwitchTo(oldcontext);
@@ src/backend/catalog/pg_publication.c: pg_get_publication_tables(PG_FUNCTION_ARGS
process_relation_publications(Oid relid, const List *publications,
PublicationActions *pubactions,
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, const List *publications,
+ ListCell *lc;
+ int publish_ancestor_level = 0;
bool am_partition = get_rel_relispartition(relid);
++ bool am_pubviaroot = false;
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
--
-- *publish_as_relid = relid;
+ Oid publish_candidate = relid;
++
++ /*
++ * For publish_via_parent handling later, it's useful to know whether the
++ * current table is explicitly part of a publish_via_partition_root
++ * publication up front.
++ */
++ foreach(lc, publications)
++ {
++ Publication *pub = lfirst(lc);
+
+- *publish_as_relid = relid;
++ if (!pub->pubviaroot)
++ continue;
++
++ if (pub->alltables ||
++ list_member_oid(pubids, pub->oid) ||
++ list_member_oid(schemaPubids, pub->oid))
++ {
++ am_pubviaroot = true;
++ break;
++ }
++ }
foreach(lc, publications)
{
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
+ }
}
}
-
- if (!publish)
+-
+- if (!publish)
++ else
{
bool ancestor_published = false;
+ Oid ancestor;
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
--
-- ancestor = GetTopMostAncestorInPublication(pub->oid,
-- ancestors,
-- &level);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
+- ancestor = GetTopMostAncestorInPublication(pub->oid,
+- ancestors,
+- &level);
+-
- if (ancestor != InvalidOid)
+ if (ancestor != InvalidOid)
+ {
@@ src/backend/catalog/pg_publication.c: process_relation_publications(Oid relid, c
if (list_member_oid(pubids, pub->oid) ||
list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
-+ (am_partition && ancestor_published))
++ (ancestor_published && am_partition) ||
++ (ancestor_published && am_pubviaroot && pub->pubviaroot))
publish = true;
}
@@ src/backend/commands/publicationcmds.c: TransformPubWhereClauses(List *tables, c
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
-+ * TODO: decide how this interacts with publish_via_parent
++ * TODO: decide how this interacts with publish_via_parent. Probably not
++ * in the same way, since logical roots can contain data of their own.
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ src/backend/commands/publicationcmds.c: CheckPubRelationColumnList(char *pubname
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
-+ * TODO: decide if this interacts with publish_via_parent
++ * TODO: decide if this interacts with publish_via_parent. Probably not
++ * (see above).
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
-+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
-+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
++CREATE PUBLICATION ipub_all FOR TABLE sch1.iroot;
++CREATE PUBLICATION ipub_one FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
-+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
-+ ONLY sch1.ipart2
++CREATE PUBLICATION ipub_two FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
-+-- At this point, the published trees look like this:
-+--
-+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
-+-- ------------------ ----------------------- -----------------------
-+-- iroot iroot
-+-- +- ipart1 ipart1 |
-+-- | +- ipart1_a | +--- ipart1_a
-+-- | +- ipart1_a1 +--- ipart1_a1 |
-+-- +- ipart2 +- ipart2
-+-- What a subscription to only ipub_root should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1
++-- | +- ipart1_a - ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++-- A subscription to only ipub_all should see every individual table.
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_all');
+ relid
+----------------
+ sch1.iroot
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_sync('ipub_root');
++SELECT * FROM published_sync('ipub_all');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
@@ src/test/regress/expected/publication.out: SELECT * FROM pg_publication_tables;
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_root');
-+ pubname | published | synced
-+-----------+----------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart2 | sch1.ipart2
-+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
-+ ipub_root | sch1.ipart1_a1 | sch1.ipart1_a1
++SELECT * FROM published_stream('ipub_all');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+-- What a subscription to only ipub_part1 should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+ relid
-+-------------
-+ sch1.ipart1
-+(1 row)
++-- A subscription to both ipub_all and ipub_one shouldn't change the initial
++-- sync, since there is no alternative root being published for ipart1_a. It
++-- will be duplicated in the stream list, since TODO
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart1
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1_a
++(6 rows)
++
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a | sch1.ipart1_a
++ sch1.ipart1_a1 | sch1.ipart1_a1
++(5 rows)
++
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.ipart1_a | sch1.ipart1_a
++(6 rows)
+
-+SELECT * FROM published_sync('ipub_part1');
-+ published | synced
-+-------------+----------------
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
-+(2 rows)
-+
-+SELECT * FROM published_stream('ipub_part1');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+(2 rows)
-+
-+-- What a subscription to both ipub_root and ipub_part1 should see
++ALTER PUBLICATION ipub_one ADD TABLE ONLY sch1.ipart1;
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1 - ipart1
++-- | +- ipart1_a +- ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++-- Adding ipart1_a's parent table to ipub_one results in ipart1_a1 being
++-- "stolen" from the ipub_all stream, to prevent its data from being duplicated.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+---------------
-+ ipub_root | sch1.iroot
-+ ipub_root | sch1.ipart1
-+ ipub_root | sch1.ipart2
-+ ipub_root | sch1.ipart1_a
-+ ipub_part1 | sch1.ipart1
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart1
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1
+(5 rows)
+
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
-+ published | synced
-+---------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
-+ sch1.ipart2 | sch1.ipart2
-+ sch1.ipart1_a | sch1.ipart1_a
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1 | sch1.ipart1_a
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
-+ pubname | published | synced
-+------------+---------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_root | sch1.ipart2 | sch1.ipart2
-+ ipub_root | sch1.ipart1_a | sch1.ipart1_a
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart1 | sch1.ipart1_a
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.ipart1 | sch1.ipart1
++ ipub_one | sch1.ipart1 | sch1.ipart1_a
+(7 rows)
+
-+-- What a subscription to both ipub_part1 and ipub_other should see
++-- Including ipub_two in the subscription list should change nothing about the
++-- sync. ipub_two will gain an entry for ipart1_a in the streaming list.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+-------------
-+ ipub_part1 | sch1.ipart1
-+ ipub_other | sch1.iroot
-+(2 rows)
-+
-+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
-+ published | synced
-+-------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.iroot | sch1.ipart2
-+ sch1.iroot | sch1.ipart1_a
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart1
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_other | sch1.iroot | sch1.iroot
-+ ipub_other | sch1.iroot | sch1.ipart2
-+ ipub_other | sch1.iroot | sch1.ipart1_a
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1 | sch1.ipart1_a
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+-- What a subscription to all three should see
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.ipart1 | sch1.ipart1
++ ipub_all | sch1.ipart1 | sch1.ipart1_a
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.ipart1 | sch1.ipart1
++ ipub_one | sch1.ipart1 | sch1.ipart1_a
++ ipub_two | sch1.ipart1 | sch1.ipart1_a
++(8 rows)
++
++ALTER PUBLICATION ipub_two ADD TABLE ONLY sch1.iroot;
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot iroot
++-- +- ipart1 - ipart1 |
++-- | +- ipart1_a +- ipart1_a +---- ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++-- Adding iroot to ipub_two ends up stealing both ipart1 and ipart1_a from the
++-- other publications, again so that no data is double-published.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+-------------
-+ ipub_root | sch1.iroot
-+ ipub_root | sch1.ipart1
-+ ipub_part1 | sch1.ipart1
-+ ipub_other | sch1.iroot
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a1
++ ipub_two | sch1.iroot
+(4 rows)
+
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+ published | synced
-+-------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.iroot | sch1.ipart2
-+ sch1.iroot | sch1.ipart1_a
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++ published | synced
++----------------+----------------
++ sch1.iroot | sch1.iroot
++ sch1.iroot | sch1.ipart1
++ sch1.iroot | sch1.ipart1_a
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.iroot | sch1.ipart2
-+ ipub_root | sch1.iroot | sch1.ipart1_a
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_other | sch1.iroot | sch1.iroot
-+ ipub_other | sch1.iroot | sch1.ipart2
-+ ipub_other | sch1.iroot | sch1.ipart1_a
-+(10 rows)
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.iroot | sch1.ipart1
++ ipub_all | sch1.iroot | sch1.ipart1_a
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.iroot | sch1.ipart1
++ ipub_one | sch1.iroot | sch1.ipart1_a
++ ipub_two | sch1.iroot | sch1.iroot
++ ipub_two | sch1.iroot | sch1.ipart1_a
++(9 rows)
+
+-- "Detaching" partitions should change the subscriptions
-+ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++ALTER TABLE sch1.ipart1_a SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+ pubname | relid
-+------------+-------------
-+ ipub_root | sch1.iroot
-+ ipub_root | sch1.ipart1
-+ ipub_root | sch1.ipart2
-+ ipub_part1 | sch1.ipart1
-+ ipub_other | sch1.iroot
-+ ipub_other | sch1.ipart2
-+(6 rows)
-+
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+ published | synced
-+-------------+----------------
-+ sch1.iroot | sch1.iroot
-+ sch1.iroot | sch1.ipart1_a
-+ sch1.ipart1 | sch1.ipart1
-+ sch1.ipart1 | sch1.ipart1_a1
-+ sch1.ipart2 | sch1.ipart2
-+(5 rows)
-+
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
-+ pubname | published | synced
-+------------+-------------+----------------
-+ ipub_root | sch1.iroot | sch1.iroot
-+ ipub_root | sch1.iroot | sch1.ipart1_a
-+ ipub_root | sch1.ipart1 | sch1.ipart1
-+ ipub_root | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_root | sch1.ipart2 | sch1.ipart2
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1_a1
-+ ipub_other | sch1.iroot | sch1.iroot
-+ ipub_other | sch1.iroot | sch1.ipart1_a
-+ ipub_other | sch1.ipart2 | sch1.ipart2
-+(10 rows)
-+
-+ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+ relid
-+----------------
-+ sch1.ipart1
-+ sch1.ipart1_a1
-+(2 rows)
++ pubname | relid
++----------+----------------
++ ipub_all | sch1.iroot
++ ipub_all | sch1.ipart2
++ ipub_all | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1
++ ipub_one | sch1.ipart1_a
++ ipub_two | sch1.iroot
++ ipub_two | sch1.ipart1_a
++(7 rows)
+
-+SELECT * FROM published_sync('ipub_part1');
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+ published | synced
+----------------+----------------
-+ sch1.ipart1 | sch1.ipart1
++ sch1.iroot | sch1.iroot
++ sch1.iroot | sch1.ipart1
++ sch1.ipart2 | sch1.ipart2
++ sch1.ipart1_a | sch1.ipart1_a
+ sch1.ipart1_a1 | sch1.ipart1_a1
-+(2 rows)
-+
-+SELECT * FROM published_stream('ipub_part1');
-+ pubname | published | synced
-+------------+----------------+----------------
-+ ipub_part1 | sch1.ipart1 | sch1.ipart1
-+ ipub_part1 | sch1.ipart1_a1 | sch1.ipart1_a1
-+(2 rows)
-+
-+DROP PUBLICATION ipub_other;
-+DROP PUBLICATION ipub_part1;
-+DROP PUBLICATION ipub_root;
++(5 rows)
++
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+----------------+----------------
++ ipub_all | sch1.iroot | sch1.iroot
++ ipub_all | sch1.iroot | sch1.ipart1
++ ipub_all | sch1.ipart2 | sch1.ipart2
++ ipub_all | sch1.ipart1_a | sch1.ipart1_a
++ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
++ ipub_one | sch1.iroot | sch1.ipart1
++ ipub_one | sch1.ipart1_a | sch1.ipart1_a
++ ipub_two | sch1.iroot | sch1.iroot
++ ipub_two | sch1.ipart1_a | sch1.ipart1_a
++(9 rows)
++
++ALTER TABLE sch1.ipart1 RESET (publish_via_parent);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_one', 'ipub_two') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++ pubname | relid
++----------+---------------
++ ipub_one | sch1.ipart1
++ ipub_one | sch1.ipart1_a
++ ipub_two | sch1.iroot
++ ipub_two | sch1.ipart1_a
++(4 rows)
++
++SELECT * FROM published_sync('ipub_one', 'ipub_two');
++ published | synced
++---------------+---------------
++ sch1.iroot | sch1.iroot
++ sch1.ipart1 | sch1.ipart1
++ sch1.ipart1_a | sch1.ipart1_a
++(3 rows)
++
++SELECT * FROM published_stream('ipub_one', 'ipub_two');
++ pubname | published | synced
++----------+---------------+---------------
++ ipub_one | sch1.ipart1 | sch1.ipart1
++ ipub_one | sch1.ipart1_a | sch1.ipart1_a
++ ipub_two | sch1.iroot | sch1.iroot
++ ipub_two | sch1.ipart1_a | sch1.ipart1_a
++(4 rows)
++
++DROP PUBLICATION ipub_all;
++DROP PUBLICATION ipub_one;
++DROP PUBLICATION ipub_two;
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
@@ src/test/regress/sql/publication.sql: ALTER TABLE sch1.tbl1 ATTACH PARTITION sch
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+
-+CREATE PUBLICATION ipub_root FOR TABLE sch1.iroot;
-+CREATE PUBLICATION ipub_part1 FOR TABLE ONLY sch1.ipart1, ONLY sch1.ipart1_a1
++CREATE PUBLICATION ipub_all FOR TABLE sch1.iroot;
++CREATE PUBLICATION ipub_one FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
-+CREATE PUBLICATION ipub_other FOR TABLE ONLY sch1.iroot, ONLY sch1.ipart1_a,
-+ ONLY sch1.ipart2
++CREATE PUBLICATION ipub_two FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
+
-+-- At this point, the published trees look like this:
-+--
-+-- ipub_root ipub_part1 (pubviaroot) ipub_other (pubviaroot)
-+-- ------------------ ----------------------- -----------------------
-+-- iroot iroot
-+-- +- ipart1 ipart1 |
-+-- | +- ipart1_a | +--- ipart1_a
-+-- | +- ipart1_a1 +--- ipart1_a1 |
-+-- +- ipart2 +- ipart2
-+
-+-- What a subscription to only ipub_root should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_root');
-+SELECT * FROM published_sync('ipub_root');
-+SELECT * FROM published_stream('ipub_root');
-+
-+-- What a subscription to only ipub_part1 should see
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+SELECT * FROM published_sync('ipub_part1');
-+SELECT * FROM published_stream('ipub_part1');
-+
-+-- What a subscription to both ipub_root and ipub_part1 should see
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1
++-- | +- ipart1_a - ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++
++-- A subscription to only ipub_all should see every individual table.
++SELECT relid::regclass FROM pg_get_publication_tables('ipub_all');
++SELECT * FROM published_sync('ipub_all');
++SELECT * FROM published_stream('ipub_all');
++
++-- A subscription to both ipub_all and ipub_one shouldn't change the initial
++-- sync, since there is no alternative root being published for ipart1_a. It
++-- will be duplicated in the stream list, since TODO
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1');
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1');
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
++
++ALTER PUBLICATION ipub_one ADD TABLE ONLY sch1.ipart1;
++
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot
++-- +- ipart1 - ipart1
++-- | +- ipart1_a +- ipart1_a - ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
+
-+-- What a subscription to both ipub_part1 and ipub_other should see
++-- Adding ipart1_a's parent table to ipub_one results in ipart1_a1 being
++-- "stolen" from the ipub_all stream, to prevent its data from being duplicated.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_part1', 'ipub_other');
-+SELECT * FROM published_stream('ipub_part1', 'ipub_other');
++SELECT * FROM published_sync('ipub_all', 'ipub_one');
++SELECT * FROM published_stream('ipub_all', 'ipub_one');
+
-+-- What a subscription to all three should see
++-- Including ipub_two in the subscription list should change nothing about the
++-- sync. ipub_two will gain an entry for ipart1_a in the streaming list.
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
++
++ALTER PUBLICATION ipub_two ADD TABLE ONLY sch1.iroot;
++
++-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
++-- ------------------ --------------------- ---------------------
++-- iroot iroot
++-- +- ipart1 - ipart1 |
++-- | +- ipart1_a +- ipart1_a +---- ipart1_a
++-- | +- ipart1_a1
++-- +- ipart2
++
++-- Adding iroot to ipub_two ends up stealing both ipart1 and ipart1_a from the
++-- other publications, again so that no data is double-published.
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+
+-- "Detaching" partitions should change the subscriptions
-+ALTER TABLE sch1.ipart2 SET (publish_via_parent = false);
++ALTER TABLE sch1.ipart1_a SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
-+ FROM pg_get_publication_tables('ipub_root', 'ipub_part1', 'ipub_other') t
++ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
-+SELECT * FROM published_sync('ipub_root', 'ipub_part1', 'ipub_other');
-+SELECT * FROM published_stream('ipub_root', 'ipub_part1', 'ipub_other');
++SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+
-+ALTER TABLE sch1.ipart1_a1 RESET (publish_via_parent);
-+SELECT relid::regclass FROM pg_get_publication_tables('ipub_part1');
-+SELECT * FROM published_sync('ipub_part1');
-+SELECT * FROM published_stream('ipub_part1');
++ALTER TABLE sch1.ipart1 RESET (publish_via_parent);
++SELECT pubname, relid::regclass
++ FROM pg_get_publication_tables('ipub_one', 'ipub_two') t
++ JOIN pg_publication p ON (p.oid = t.pubid);
++SELECT * FROM published_sync('ipub_one', 'ipub_two');
++SELECT * FROM published_stream('ipub_one', 'ipub_two');
+
-+DROP PUBLICATION ipub_other;
-+DROP PUBLICATION ipub_part1;
-+DROP PUBLICATION ipub_root;
++DROP PUBLICATION ipub_all;
++DROP PUBLICATION ipub_one;
++DROP PUBLICATION ipub_two;
+
RESET client_min_messages;
DROP PUBLICATION pub;
[text/x-patch] v4-0001-pgoutput-refactor-publication-cache-construction.patch (18.1K, ../../CAAWbhmjcnoV7Xu6LHr_hxqWmVtehv404bvDye+QZcUDSg8NSKw@mail.gmail.com/3-v4-0001-pgoutput-refactor-publication-cache-construction.patch)
download | inline diff:
From 173e74279c22193346b91226dce9d6a8273b9f60 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Fri, 7 Apr 2023 09:55:27 -0700
Subject: [PATCH v4 1/2] pgoutput: refactor publication cache construction
Breaking this logic into a standalone helper should help expose the
behavior for testing. Additionally, move the implementation under the
pg_publication catalog helpers, since it seems like it's inherent to the
publication settings (and it must match the behavior of the initial
sync, which doesn't seem to be controlled by the output plugin at all).
---
src/backend/catalog/pg_publication.c | 215 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 137 +------------
src/include/catalog/pg_proc.dat | 8 +
src/include/catalog/pg_publication.h | 4 +
src/test/regress/expected/publication.out | 26 +++
src/test/regress/sql/publication.sql | 15 ++
6 files changed, 272 insertions(+), 133 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index c488b6370b..df9abf09c3 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -1259,3 +1259,218 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+List *
+process_relation_publications(Oid relid, const List *publications,
+ PublicationActions *pubactions,
+ Oid *publish_as_relid)
+{
+ Oid schemaId = get_rel_namespace(relid);
+ List *pubids = GetRelationPublications(relid);
+
+ /*
+ * We don't acquire a lock on the namespace system table as we build
+ * the cache entry using a historic snapshot and all the later changes
+ * are absorbed while decoding WAL.
+ */
+ List *schemaPubids = GetSchemaPublications(schemaId);
+ ListCell *lc;
+ int publish_ancestor_level = 0;
+ bool am_partition = get_rel_relispartition(relid);
+ char relkind = get_rel_relkind(relid);
+ List *rel_publications = NIL;
+
+ *publish_as_relid = relid;
+
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ bool publish = false;
+
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications. Also
+ * track the ancestor level for this publication.
+ */
+ Oid pub_relid = relid;
+ int ancestor_level = 0;
+
+ /*
+ * If this is a FOR ALL TABLES publication, pick the partition
+ * root and set the ancestor level accordingly.
+ */
+ if (pub->alltables)
+ {
+ publish = true;
+ if (pub->pubviaroot && am_partition)
+ {
+ List *ancestors = get_partition_ancestors(relid);
+
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
+ }
+
+ if (!publish)
+ {
+ bool ancestor_published = false;
+
+ /*
+ * For a partition, check if any of the ancestors are
+ * published. If so, note down the topmost ancestor that is
+ * published via this publication, which will be used as the
+ * relation via which to publish the partition's changes.
+ */
+ if (am_partition)
+ {
+ Oid ancestor;
+ int level;
+ List *ancestors = get_partition_ancestors(relid);
+
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
+
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
+ {
+ pub_relid = ancestor;
+ ancestor_level = level;
+ }
+ }
+ }
+
+ if (list_member_oid(pubids, pub->oid) ||
+ list_member_oid(schemaPubids, pub->oid) ||
+ ancestor_published)
+ publish = true;
+ }
+
+ /*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
+ * Don't publish changes for partitioned tables, because
+ * publishing those of its partitions suffices, unless partition
+ * changes won't be published due to pubviaroot being set.
+ */
+ if (publish &&
+ (relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
+ {
+ pubactions->pubinsert |= pub->pubactions.pubinsert;
+ pubactions->pubupdate |= pub->pubactions.pubupdate;
+ pubactions->pubdelete |= pub->pubactions.pubdelete;
+ pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we need to check if the already
+ * calculated level is higher than the new one. If yes, we can
+ * ignore the new value (as it's a child). Otherwise the new
+ * value is an ancestor, so we keep it.
+ */
+ if (publish_ancestor_level > ancestor_level)
+ continue;
+
+ /*
+ * If we found an ancestor higher up in the tree, discard the
+ * list of publications through which we replicate it, and use
+ * the new ancestor.
+ */
+ if (publish_ancestor_level < ancestor_level)
+ {
+ *publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
+
+ /* reset the publication list for this relation */
+ rel_publications = NIL;
+ }
+ else
+ {
+ /* Same ancestor level, has to be the same OID. */
+ Assert(*publish_as_relid == pub_relid);
+ }
+
+ /* Track publications for this ancestor. */
+ rel_publications = lappend(rel_publications, pub);
+ }
+ }
+
+ list_free(pubids);
+ list_free(schemaPubids);
+
+ return rel_publications;
+}
+
+Datum
+pg_get_relation_publishing_info(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ List *publications = NIL;
+ ArrayType *arr;
+ Datum *elems;
+ int nelems,
+ i;
+ TupleDesc tupdesc;
+ HeapTuple htup;
+
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ arr = PG_GETARG_ARRAYTYPE_P(1);
+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT, &elems, NULL,
+ &nelems);
+
+ /* Get Oids of tables from each publication. */
+ for (i = 0; i < nelems; i++)
+ {
+ char *pubname = TextDatumGetCString(elems[i]);
+ Publication *pub = GetPublicationByName(pubname, false);
+
+ publications = lappend(publications, pub);
+ }
+
+ {
+ List *rel_publications;
+ PublicationActions pubactions;
+ Oid publish_as_relid;
+ ListCell *lc;
+ Datum *puboids;
+ ArrayType *puboidarray;
+ Datum values[6];
+ bool nulls[6];
+
+ rel_publications =
+ process_relation_publications(relid, publications, &pubactions,
+ &publish_as_relid);
+
+ /* Translate the rel_publications List into an OID array. */
+ puboids = palloc(sizeof(Datum) * list_length(rel_publications));
+
+ foreach(lc, rel_publications)
+ {
+ Publication *pub = lfirst(lc);
+
+ i = foreach_current_index(lc);
+ puboids[i] = ObjectIdGetDatum(pub->oid);
+ }
+
+ puboidarray = construct_array(puboids, list_length(rel_publications),
+ OIDOID, sizeof(Oid), true, TYPALIGN_INT);
+
+ values[0] = PointerGetDatum(puboidarray);
+ values[1] = ObjectIdGetDatum(publish_as_relid);
+ values[2] = BoolGetDatum(pubactions.pubinsert);
+ values[3] = BoolGetDatum(pubactions.pubupdate);
+ values[4] = BoolGetDatum(pubactions.pubdelete);
+ values[5] = BoolGetDatum(pubactions.pubtruncate);
+
+ memset(nulls, false, sizeof(nulls));
+
+ htup = heap_form_tuple(tupdesc, values, nulls);
+ }
+
+ PG_RETURN_DATUM(HeapTupleGetDatum(htup));
+}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index b08ca55041..640676e7dc 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1986,20 +1986,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Validate the entry */
if (!entry->replicate_valid)
{
- Oid schemaId = get_rel_namespace(relid);
- List *pubids = GetRelationPublications(relid);
-
- /*
- * We don't acquire a lock on the namespace system table as we build
- * the cache entry using a historic snapshot and all the later changes
- * are absorbed while decoding WAL.
- */
- List *schemaPubids = GetSchemaPublications(schemaId);
- ListCell *lc;
- Oid publish_as_relid = relid;
- int publish_ancestor_level = 0;
- bool am_partition = get_rel_relispartition(relid);
- char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
/* Reload publications if needed before use. */
@@ -2063,123 +2049,10 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
* but here we only need to consider ones that the subscriber
* requested.
*/
- foreach(lc, data->publications)
- {
- Publication *pub = lfirst(lc);
- bool publish = false;
-
- /*
- * Under what relid should we publish changes in this publication?
- * We'll use the top-most relid across all publications. Also
- * track the ancestor level for this publication.
- */
- Oid pub_relid = relid;
- int ancestor_level = 0;
-
- /*
- * If this is a FOR ALL TABLES publication, pick the partition
- * root and set the ancestor level accordingly.
- */
- if (pub->alltables)
- {
- publish = true;
- if (pub->pubviaroot && am_partition)
- {
- List *ancestors = get_partition_ancestors(relid);
-
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
- }
- }
-
- if (!publish)
- {
- bool ancestor_published = false;
-
- /*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
- * published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
- */
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
-
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
-
- if (ancestor != InvalidOid)
- {
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
- }
- }
-
- if (list_member_oid(pubids, pub->oid) ||
- list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
- publish = true;
- }
-
- /*
- * If the relation is to be published, determine actions to
- * publish, and list of columns, if appropriate.
- *
- * Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
- */
- if (publish &&
- (relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
- {
- entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
- entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
- entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
- entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
-
- /*
- * We want to publish the changes as the top-most ancestor
- * across all publications. So we need to check if the already
- * calculated level is higher than the new one. If yes, we can
- * ignore the new value (as it's a child). Otherwise the new
- * value is an ancestor, so we keep it.
- */
- if (publish_ancestor_level > ancestor_level)
- continue;
-
- /*
- * If we found an ancestor higher up in the tree, discard the
- * list of publications through which we replicate it, and use
- * the new ancestor.
- */
- if (publish_ancestor_level < ancestor_level)
- {
- publish_as_relid = pub_relid;
- publish_ancestor_level = ancestor_level;
-
- /* reset the publication list for this relation */
- rel_publications = NIL;
- }
- else
- {
- /* Same ancestor level, has to be the same OID. */
- Assert(publish_as_relid == pub_relid);
- }
-
- /* Track publications for this ancestor. */
- rel_publications = lappend(rel_publications, pub);
- }
- }
-
- entry->publish_as_relid = publish_as_relid;
+ rel_publications =
+ process_relation_publications(relid, data->publications,
+ &entry->pubactions,
+ &entry->publish_as_relid);
/*
* Initialize the tuple slot, map, and row filter. These are only used
@@ -2198,8 +2071,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
pgoutput_column_list_init(data, rel_publications, entry);
}
- list_free(pubids);
- list_free(schemaPubids);
list_free(rel_publications);
entry->replicate_valid = true;
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9805bc6118..0598af0cf9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11852,6 +11852,14 @@
proname => 'pg_relation_is_publishable', provolatile => 's',
prorettype => 'bool', proargtypes => 'regclass',
prosrc => 'pg_relation_is_publishable' },
+{ oid => '8139',
+ descr => 'get information on how a relation will be published via a list of publications',
+ proname => 'pg_get_relation_publishing_info', provariadic => 'text',
+ provolatile => 's', prorettype => 'record', proargtypes => 'regclass _text',
+ proallargtypes => '{regclass,_text,_oid,oid,bool,bool,bool,bool}',
+ proargmodes => '{i,v,o,o,o,o,o,o}',
+ proargnames => '{relid,pubnames,pubids,pubasrelid,pubinsert,pubupdate,pubdelete,pubtruncate}',
+ prosrc => 'pg_get_relation_publishing_info' },
# rls
{ oid => '3298',
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 6ecaa2a01e..d4e3488917 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -155,4 +155,8 @@ extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
MemoryContext mcxt);
+extern List *process_relation_publications(Oid relid, const List *publications,
+ PublicationActions *pubactions,
+ Oid *publish_as_relid);
+
#endif /* PG_PUBLICATION_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 16361a91f9..85c217dadb 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -5,6 +5,17 @@ CREATE ROLE regress_publication_user LOGIN SUPERUSER;
CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_stream (VARIADIC pubnames text[])
+ RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
+ -- For each publication, show each published root alongside the tables which
+ -- are published via its OID.
+ SELECT p.pubname, rpi.pubasrelid::regclass, pr.prrelid::regclass
+ FROM pg_publication_rel pr
+ JOIN pg_publication p ON (p.oid = pr.prpubid),
+ pg_get_relation_publishing_info(pr.prrelid, VARIADIC pubnames) rpi
+ WHERE p.pubname = ANY (pubnames)
+ ORDER BY p.oid, 2, 3;
+$$ LANGUAGE sql;
-- suppress warning that depends on wal_level
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_default;
@@ -1692,6 +1703,13 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+SELECT * FROM published_stream('pub');
+ pubname | published | synced
+---------+-----------+-----------------
+ pub | sch1.tbl1 | sch1.tbl1
+ pub | sch1.tbl1 | sch2.tbl1_part1
+(2 rows)
+
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
@@ -1718,6 +1736,13 @@ SELECT * FROM pg_publication_tables;
pub | sch2 | tbl1_part1 | {a} |
(1 row)
+SELECT * FROM published_stream('pub');
+ pubname | published | synced
+---------+-----------------+-----------------
+ pub | sch1.tbl1 | sch1.tbl1
+ pub | sch2.tbl1_part1 | sch2.tbl1_part1
+(2 rows)
+
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
DROP TABLE sch1.tbl1;
@@ -1738,6 +1763,7 @@ DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
+DROP FUNCTION published_stream;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
DROP ROLE regress_publication_user_dummy;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index d5051a5e74..6bf9554d48 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -6,6 +6,18 @@ CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_stream (VARIADIC pubnames text[])
+ RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
+ -- For each publication, show each published root alongside the tables which
+ -- are published via its OID.
+ SELECT p.pubname, rpi.pubasrelid::regclass, pr.prrelid::regclass
+ FROM pg_publication_rel pr
+ JOIN pg_publication p ON (p.oid = pr.prpubid),
+ pg_get_relation_publishing_info(pr.prrelid, VARIADIC pubnames) rpi
+ WHERE p.pubname = ANY (pubnames)
+ ORDER BY p.oid, 2, 3;
+$$ LANGUAGE sql;
+
-- suppress warning that depends on wal_level
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_default;
@@ -1064,6 +1076,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
@@ -1078,6 +1091,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
@@ -1096,6 +1110,7 @@ DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
+DROP FUNCTION published_stream;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
--
2.34.1
[text/x-patch] v4-0002-WIP-introduce-publish_via_parent-for-logical-repl.patch (87.4K, ../../CAAWbhmjcnoV7Xu6LHr_hxqWmVtehv404bvDye+QZcUDSg8NSKw@mail.gmail.com/4-v4-0002-WIP-introduce-publish_via_parent-for-logical-repl.patch)
download | inline diff:
From dc2425ceaa8dd4fe68d1b1dc780de15800041564 Mon Sep 17 00:00:00 2001
From: Jacob Champion <[email protected]>
Date: Mon, 26 Sep 2022 13:23:51 -0700
Subject: [PATCH v4 2/2] WIP: introduce publish_via_parent for logical
replication
This new relation option allows regular inherited tables to be published
via their root table, just like partitions. This works by hijacking
pg_inherit's inhseqno column, and replacing a (single) existing entry
for the child with the value zero, indicating that it should be treated
as a logical partition by the publication machinery.
Initial sync works by asking the publisher for a list of logical
descendants of the published table, then COPYing them one-by-one into
the root. The publisher reuses the existing pubviaroot logic, adding the
new logical roots to code that previously looked only for partition
roots.
Known bugs/TODOs:
- I haven't given any thought to interactions with row filters, or to
column lists.
- I'm not sure that I'm taking all the necessary locks yet, and those I
do take may be taken in the wrong order.
---
src/backend/access/common/reloptions.c | 11 +
src/backend/catalog/pg_inherits.c | 123 ++++++-
src/backend/catalog/pg_publication.c | 336 +++++++++++++-----
src/backend/commands/publicationcmds.c | 12 +
src/backend/commands/tablecmds.c | 161 ++++++++-
src/backend/partitioning/partdesc.c | 3 +-
src/backend/replication/logical/tablesync.c | 257 +++++++++-----
src/backend/replication/pgoutput/pgoutput.c | 1 -
src/bin/pg_dump/t/002_pg_dump.pl | 24 ++
src/bin/psql/tab-complete.c | 1 +
src/include/catalog/pg_inherits.h | 8 +-
src/include/catalog/pg_proc.dat | 7 +
src/include/utils/rel.h | 12 +
src/test/regress/expected/publication.out | 359 ++++++++++++++++++++
src/test/regress/sql/publication.sql | 166 +++++++++
src/test/subscription/t/013_partition.pl | 312 +++++++++++++++++
16 files changed, 1613 insertions(+), 180 deletions(-)
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 469de9bb49..1ef081b78d 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -122,6 +122,15 @@ static relopt_bool boolRelOpts[] =
},
false
},
+ {
+ {
+ "publish_via_parent",
+ "Replicate this table's contents via its parent table when publishing with publish_via_partition_root",
+ RELOPT_KIND_HEAP,
+ AccessExclusiveLock
+ },
+ false
+ },
{
{
"fastupdate",
@@ -1877,6 +1886,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind)
offsetof(StdRdOptions, autovacuum) + offsetof(AutoVacOpts, analyze_scale_factor)},
{"user_catalog_table", RELOPT_TYPE_BOOL,
offsetof(StdRdOptions, user_catalog_table)},
+ {"publish_via_parent", RELOPT_TYPE_BOOL,
+ offsetof(StdRdOptions, publish_via_parent)},
{"parallel_workers", RELOPT_TYPE_INT,
offsetof(StdRdOptions, parallel_workers)},
{"vacuum_index_cleanup", RELOPT_TYPE_ENUM,
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index da969bd2f9..cf90769457 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -24,14 +24,22 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_inherits.h"
+#include "catalog/partition.h"
+#include "miscadmin.h"
#include "parser/parse_type.h"
#include "storage/lmgr.h"
+#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/fmgrprotos.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+static List * find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only);
+
/*
* Entry of a hash table used in find_all_inheritors. See below.
*/
@@ -59,14 +67,14 @@ List *
find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
{
return find_inheritance_children_extended(parentrelId, true, lockmode,
- NULL, NULL);
+ NULL, NULL, false);
}
/*
* find_inheritance_children_extended
*
* As find_inheritance_children, with more options regarding detached
- * partitions.
+ * partitions and logical roots.
*
* If a partition's pg_inherits row is marked "detach pending",
* *detached_exist (if not null) is set true.
@@ -78,16 +86,21 @@ find_inheritance_children(Oid parentrelId, LOCKMODE lockmode)
* whether the transaction that marked those partitions as detached appears
* committed to the active snapshot. In addition, *detached_xmin (if not null)
* is set to the xmin of the row of the detached partition.
+ *
+ * If logical_only is true, only tables marked explicitly via publish_via_parent
+ * are included in the output list.
*/
List *
find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
LOCKMODE lockmode, bool *detached_exist,
- TransactionId *detached_xmin)
+ TransactionId *detached_xmin,
+ bool logical_only)
{
List *list = NIL;
Relation relation;
+ Oid index;
SysScanDesc scan;
- ScanKeyData key[1];
+ ScanKeyData key[2];
HeapTuple inheritsTuple;
Oid inhrelid;
Oid *oidarr;
@@ -110,14 +123,24 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
numoids = 0;
relation = table_open(InheritsRelationId, AccessShareLock);
+ index = InheritsParentIndexId;
ScanKeyInit(&key[0],
Anum_pg_inherits_inhparent,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(parentrelId));
- scan = systable_beginscan(relation, InheritsParentIndexId, true,
- NULL, 1, key);
+ if (logical_only)
+ {
+ index = InheritsParentSeqnoIndexId;
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+ }
+
+ scan = systable_beginscan(relation, index, true,
+ NULL, logical_only ? 2 : 1, key);
while ((inheritsTuple = systable_getnext(scan)) != NULL)
{
@@ -254,6 +277,20 @@ find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
*/
List *
find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
+{
+ return find_all_inheritors_internal(parentrelId, lockmode, numparents,
+ false);
+}
+
+List *
+find_all_logical_inheritors(Oid parentrelId)
+{
+ return find_all_inheritors_internal(parentrelId, NoLock, NULL, true);
+}
+
+static List *
+find_all_inheritors_internal(Oid parentrelId, LOCKMODE lockmode,
+ List **numparents, bool logical_only)
{
/* hash table for O(1) rel_oid -> rel_numparents cell lookup */
HTAB *seen_rels;
@@ -290,7 +327,9 @@ find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents)
ListCell *lc;
/* Get the direct children of this rel */
- currentchildren = find_inheritance_children(currentrel, lockmode);
+ currentchildren =
+ find_inheritance_children_extended(currentrel, true, lockmode,
+ NULL, NULL, logical_only);
/*
* Add to the queue only those children not already seen. This avoids
@@ -655,3 +694,73 @@ PartitionHasPendingDetach(Oid partoid)
elog(ERROR, "relation %u is not a partition", partoid);
return false; /* keep compiler quiet */
}
+
+static Oid
+get_logical_parent_worker(Relation inhRel, Oid relid)
+{
+ SysScanDesc scan;
+ ScanKeyData key[2];
+ Oid result = InvalidOid;
+ HeapTuple tuple;
+
+ ScanKeyInit(&key[0],
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+ ScanKeyInit(&key[1],
+ Anum_pg_inherits_inhseqno,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(0));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 2, key);
+ tuple = systable_getnext(scan);
+ if (HeapTupleIsValid(tuple))
+ {
+ Form_pg_inherits form = (Form_pg_inherits) GETSTRUCT(tuple);
+ result = form->inhparent;
+ }
+
+ systable_endscan(scan);
+
+ return result;
+}
+
+static void
+get_logical_ancestors_worker(Relation inhRel, Oid relid, List **ancestors)
+{
+ Oid parentOid;
+
+ /*
+ * Recursion ends at the topmost level, ie., when there's no parent.
+ */
+ parentOid = get_logical_parent_worker(inhRel, relid);
+ if (parentOid == InvalidOid)
+ return;
+
+ *ancestors = lappend_oid(*ancestors, parentOid);
+ get_logical_ancestors_worker(inhRel, parentOid, ancestors);
+}
+
+List *
+get_logical_ancestors(Oid relid, bool is_partition)
+{
+ List *result = NIL;
+ Relation inhRel;
+
+ /* For partitions, this is identical to get_partition_ancestors(). */
+ if (is_partition)
+ return get_partition_ancestors(relid);
+
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
+ get_logical_ancestors_worker(inhRel, relid, &result);
+ table_close(inhRel, AccessShareLock);
+
+ return result;
+}
+
+bool
+has_logical_parent(Relation inhRel, Oid relid)
+{
+ return (get_logical_parent_worker(inhRel, relid) != InvalidOid);
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index df9abf09c3..20970a4eb0 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -51,6 +51,7 @@ typedef struct
Oid relid; /* OID of published table */
Oid pubid; /* OID of publication that publishes this
* table. */
+ bool viaroot; /* does pubid use publish_via_partition_root? */
} published_rel;
static void publication_translate_columns(Relation targetrel, List *columns,
@@ -180,56 +181,46 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
}
/*
- * Returns true if the ancestor is in the list of published relations.
- * Otherwise, returns false.
- */
-static bool
-is_ancestor_member_tableinfos(Oid ancestor, List *table_infos)
-{
- ListCell *lc;
-
- foreach(lc, table_infos)
- {
- Oid relid = ((published_rel *) lfirst(lc))->relid;
-
- if (relid == ancestor)
- return true;
- }
-
- return false;
-}
-
-/*
- * Filter out the partitions whose parent tables are also present in the list.
+ * Filter out the tables who are already being published via a logical root.
+ *
+ * If we only had partitioned tables to check, this would be much easier: it's
+ * impossible for a partition root to be in this list unless pubviaroot=true for
+ * that table, so we would just have to check for the existence of an ancestor,
+ * and if any were found then we'd filter the table.
+ *
+ * Logical roots add another wrinkle, because it's acceptable for a
+ * pubviaroot=false publication to contain a logical root. (Logical roots can
+ * contain data of their own, unlike partition roots.) It's also possible for a
+ * logical root to be included in a pubviaroot publication without any or all of
+ * its children. This is not possible for a partition root, where all of the
+ * leaves are implicitly included.
+ *
+ * So, to keep things relatively consistent, this now uses the same code that
+ * the output plugin uses to decide which relation OID to publish data under. If
+ * the publishing OID is not the same as the relation OID, we remove it from
+ * this list.
+ *
+ * XXX And that's expensive.
*/
static void
-filter_partitions(List *table_infos)
+filter_published_descendants(List *table_infos, List *publications)
{
ListCell *lc;
foreach(lc, table_infos)
{
- bool skip = false;
- List *ancestors = NIL;
- ListCell *lc2;
published_rel *table_info = (published_rel *) lfirst(lc);
+ Oid publish_as_relid;
+ List *rel_publications;
- if (get_rel_relispartition(table_info->relid))
- ancestors = get_partition_ancestors(table_info->relid);
-
- foreach(lc2, ancestors)
- {
- Oid ancestor = lfirst_oid(lc2);
-
- if (is_ancestor_member_tableinfos(ancestor, table_infos))
- {
- skip = true;
- break;
- }
- }
+ rel_publications =
+ process_relation_publications(table_info->relid, publications, NULL,
+ &publish_as_relid);
- if (skip)
+ if (publish_as_relid != table_info->relid)
table_infos = foreach_delete_current(table_infos, lc);
+
+ list_free(rel_publications);
}
}
@@ -805,11 +796,15 @@ List *
GetAllTablesPublicationRelations(bool pubviaroot)
{
Relation classRel;
+ Relation inhRel;
ScanKeyData key[1];
TableScanDesc scan;
HeapTuple tuple;
List *result = NIL;
+ /* TODO: is there a required order to acquire these locks? */
+ if (pubviaroot)
+ inhRel = table_open(InheritsRelationId, AccessShareLock);
classRel = table_open(RelationRelationId, AccessShareLock);
ScanKeyInit(&key[0],
@@ -825,7 +820,8 @@ GetAllTablesPublicationRelations(bool pubviaroot)
Oid relid = relForm->oid;
if (is_publishable_class(relid, relForm) &&
- !(relForm->relispartition && pubviaroot))
+ !(relForm->relispartition && pubviaroot) &&
+ !(pubviaroot && has_logical_parent(inhRel, relid)))
result = lappend_oid(result, relid);
}
@@ -854,6 +850,9 @@ GetAllTablesPublicationRelations(bool pubviaroot)
}
table_close(classRel, AccessShareLock);
+ if (pubviaroot)
+ table_close(inhRel, AccessShareLock);
+
return result;
}
@@ -1070,6 +1069,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
int nelems,
i;
bool viaroot = false;
+ List *publications = NIL;
/* create a function context for cross-call persistence */
funcctx = SRF_FIRSTCALL_INIT();
@@ -1093,6 +1093,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
ListCell *lc;
pub_elem = GetPublicationByName(TextDatumGetCString(elems[i]), false);
+ publications = lappend(publications, pub_elem);
/*
* Publications support partitioned tables. If
@@ -1132,6 +1133,7 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
table_info->relid = lfirst_oid(lc);
table_info->pubid = pub_elem->oid;
+ table_info->viaroot = pub_elem->pubviaroot;
table_infos = lappend(table_infos, table_info);
}
@@ -1141,15 +1143,14 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
}
/*
- * If the publication publishes partition changes via their respective
- * root partitioned tables, we must exclude partitions in favor of
- * including the root partitioned tables. Otherwise, the function
- * could return both the child and parent tables which could cause
- * data of the child table to be double-published on the subscriber
- * side.
+ * If the publication publishes table changes via their respective
+ * logical root tables, we must exclude partitions in favor of
+ * including the root tables. Otherwise, the function could return
+ * both the child and parent tables which could cause data of the
+ * child table to be double-published on the subscriber side.
*/
if (viaroot)
- filter_partitions(table_infos);
+ filter_published_descendants(table_infos, publications);
/* Construct a tuple descriptor for the result rows. */
tupdesc = CreateTemplateTupleDesc(NUM_PUBLICATION_TABLES_ELEM);
@@ -1165,6 +1166,8 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
funcctx->tuple_desc = BlessTupleDesc(tupdesc);
funcctx->user_fctx = (void *) table_infos;
+ list_free(publications);
+
MemoryContextSwitchTo(oldcontext);
}
@@ -1260,6 +1263,146 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+/*
+ * Returns a list of tables that should be copied during initial sync for the
+ * given root table and publications.
+ *
+ * The only time this list consists of anything more than the table itself is
+ * when a publication's publish_via_partition_root is set to true and the table
+ * has inherited child tables in the publications that have been marked with
+ * publish_via_parent.
+ */
+Datum
+pg_get_publication_rels_to_sync(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ List *tables = NIL;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ Oid rootid = PG_GETARG_OID(0);
+ MemoryContext oldcontext;
+ List *publications = NIL;
+ List *inheritors;
+ List *candidates = NIL;
+ ListCell *lc;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ /* Construct the list of publications to consider. */
+ {
+ ArrayType *arr;
+ Datum *elems;
+ int nelems,
+ i;
+
+ /*
+ * Deconstruct the parameter into elements where each element is a
+ * publication name.
+ */
+ arr = PG_GETARG_ARRAYTYPE_P(1);
+ deconstruct_array(arr, TEXTOID, -1, false, TYPALIGN_INT,
+ &elems, NULL, &nelems);
+
+ for (i = 0; i < nelems; i++)
+ {
+ char *pubname = TextDatumGetCString(elems[i]);
+ Publication *pub = GetPublicationByName(pubname, false);
+
+ publications = lappend(publications, pub);
+ }
+ }
+
+ /* TODO: do the tables in this list need to be locked? */
+ inheritors = find_all_logical_inheritors(rootid);
+
+ /*
+ * For a logical descendant to be in the list to sync, both it and the
+ * root table need to be part of a pubviaroot publication. (But they
+ * don't have to be part of the *same* pubviaroot publication. This is
+ * somewhat counterintuitive, but it prevents a descendant from being
+ * double-published through two separate roots.)
+ */
+ foreach(lc, publications)
+ {
+ Publication *publication = lfirst(lc);
+ List *relids;
+ List *schemarelids;
+ List *published;
+ ListCell *cell;
+
+ if (!publication->pubviaroot)
+ continue;
+
+ if (publication->alltables)
+ {
+ /* Easy case: every inheritor is published through the root. */
+ candidates = inheritors;
+ break;
+ }
+
+ /*
+ * Figure out which of the inheritors is included in this
+ * publication. They get added to our candidates list.
+ */
+ relids = GetPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_ALL);
+ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_LEAF);
+ published = list_concat_unique_oid(relids, schemarelids);
+
+ foreach(cell, inheritors)
+ {
+ Oid current = lfirst_oid(cell);
+
+ if (list_member_oid(published, current))
+ candidates = list_append_unique_oid(candidates, current);
+ }
+ }
+
+ /*
+ * Finally, if we didn't find the root table itself in a pubviaroot
+ * publication, none of the logical descendants will be published
+ * through it.
+ *
+ * XXX: this assumes that the root table was part of at least one of the
+ * publications in the list. It doesn't make any sense to call this
+ * function otherwise, but should we ERROR out if that assumption is
+ * violated?
+ */
+ if (list_member_oid(candidates, rootid))
+ tables = candidates;
+ else
+ tables = list_make1_oid(rootid);
+
+ funcctx->user_fctx = (void *) tables;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+ tables = (List *) funcctx->user_fctx;
+
+ if (funcctx->call_cntr < list_length(tables))
+ {
+ Oid relid = list_nth_oid(tables, funcctx->call_cntr);
+
+ SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ }
+
+ SRF_RETURN_DONE(funcctx);
+}
+
List *
process_relation_publications(Oid relid, const List *publications,
PublicationActions *pubactions,
@@ -1277,10 +1420,31 @@ process_relation_publications(Oid relid, const List *publications,
ListCell *lc;
int publish_ancestor_level = 0;
bool am_partition = get_rel_relispartition(relid);
+ bool am_pubviaroot = false;
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
+ Oid publish_candidate = relid;
+
+ /*
+ * For publish_via_parent handling later, it's useful to know whether the
+ * current table is explicitly part of a publish_via_partition_root
+ * publication up front.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
- *publish_as_relid = relid;
+ if (!pub->pubviaroot)
+ continue;
+
+ if (pub->alltables ||
+ list_member_oid(pubids, pub->oid) ||
+ list_member_oid(schemaPubids, pub->oid))
+ {
+ am_pubviaroot = true;
+ break;
+ }
+ }
foreach(lc, publications)
{
@@ -1296,55 +1460,57 @@ process_relation_publications(Oid relid, const List *publications,
int ancestor_level = 0;
/*
- * If this is a FOR ALL TABLES publication, pick the partition
+ * If this is a FOR ALL TABLES publication, pick the logical
* root and set the ancestor level accordingly.
*/
if (pub->alltables)
{
publish = true;
- if (pub->pubviaroot && am_partition)
+ if (pub->pubviaroot)
{
- List *ancestors = get_partition_ancestors(relid);
+ List *ancestors;
- pub_relid = llast_oid(ancestors);
- ancestor_level = list_length(ancestors);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ if (ancestors != NIL)
+ {
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
}
-
- if (!publish)
+ else
{
bool ancestor_published = false;
+ Oid ancestor;
+ int level;
+ List *ancestors;
/*
- * For a partition, check if any of the ancestors are
- * published. If so, note down the topmost ancestor that is
+ * Check if any of the logical ancestors (that is, partition
+ * parents or tables marked with publish_via_parent) are
+ * published. If so, note down the topmost ancestor that is
* published via this publication, which will be used as the
- * relation via which to publish the partition's changes.
+ * relation via which to publish this table's changes.
*/
- if (am_partition)
- {
- Oid ancestor;
- int level;
- List *ancestors = get_partition_ancestors(relid);
+ ancestors = get_logical_ancestors(relid, am_partition);
+ ancestor = GetTopMostAncestorInPublication(pub->oid,
+ ancestors,
+ &level);
- ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors,
- &level);
-
- if (ancestor != InvalidOid)
+ if (ancestor != InvalidOid)
+ {
+ ancestor_published = true;
+ if (pub->pubviaroot)
{
- ancestor_published = true;
- if (pub->pubviaroot)
- {
- pub_relid = ancestor;
- ancestor_level = level;
- }
+ pub_relid = ancestor;
+ ancestor_level = level;
}
}
if (list_member_oid(pubids, pub->oid) ||
list_member_oid(schemaPubids, pub->oid) ||
- ancestor_published)
+ (ancestor_published && am_partition) ||
+ (ancestor_published && am_pubviaroot && pub->pubviaroot))
publish = true;
}
@@ -1359,10 +1525,13 @@ process_relation_publications(Oid relid, const List *publications,
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
- pubactions->pubinsert |= pub->pubactions.pubinsert;
- pubactions->pubupdate |= pub->pubactions.pubupdate;
- pubactions->pubdelete |= pub->pubactions.pubdelete;
- pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+ if (pubactions)
+ {
+ pubactions->pubinsert |= pub->pubactions.pubinsert;
+ pubactions->pubupdate |= pub->pubactions.pubupdate;
+ pubactions->pubdelete |= pub->pubactions.pubdelete;
+ pubactions->pubtruncate |= pub->pubactions.pubtruncate;
+ }
/*
* We want to publish the changes as the top-most ancestor
@@ -1381,7 +1550,7 @@ process_relation_publications(Oid relid, const List *publications,
*/
if (publish_ancestor_level < ancestor_level)
{
- *publish_as_relid = pub_relid;
+ publish_candidate = pub_relid;
publish_ancestor_level = ancestor_level;
/* reset the publication list for this relation */
@@ -1390,7 +1559,7 @@ process_relation_publications(Oid relid, const List *publications,
else
{
/* Same ancestor level, has to be the same OID. */
- Assert(*publish_as_relid == pub_relid);
+ Assert(publish_candidate == pub_relid);
}
/* Track publications for this ancestor. */
@@ -1401,6 +1570,9 @@ process_relation_publications(Oid relid, const List *publications,
list_free(pubids);
list_free(schemaPubids);
+ if (publish_as_relid)
+ *publish_as_relid = publish_candidate;
+
return rel_publications;
}
@@ -1434,7 +1606,7 @@ pg_get_relation_publishing_info(PG_FUNCTION_ARGS)
{
List *rel_publications;
- PublicationActions pubactions;
+ PublicationActions pubactions = {0};
Oid publish_as_relid;
ListCell *lc;
Datum *puboids;
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f4ba572697..a7f66d2982 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -238,6 +238,8 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* parent table, but the bitmap contains the replica identity
* information of the child table. So, get the column number of the
* child table as parent and child column order could be different.
+ *
+ * TODO: is this applicable to publish_via_parent?
*/
if (context->pubviaroot)
{
@@ -286,6 +288,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
*
* Note that even though the row filter used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
+ *
+ * TODO: is this applicable to publish_via_parent?
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
@@ -336,6 +340,8 @@ pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
* the column list.
*
* Returns true if any replica identity column is not covered by column list.
+ *
+ * TODO: publish_via_parent?
*/
bool
pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
@@ -628,6 +634,9 @@ TransformPubWhereClauses(List *tables, const char *queryString,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's row filter will be used. So disallow using
* WHERE clause on partitioned table in this case.
+ *
+ * TODO: decide how this interacts with publish_via_parent. Probably not
+ * in the same way, since logical roots can contain data of their own.
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
@@ -715,6 +724,9 @@ CheckPubRelationColumnList(char *pubname, List *tables,
* If the publication doesn't publish changes via the root partitioned
* table, the partition's column list will be used. So disallow using
* a column list on the partitioned table in this case.
+ *
+ * TODO: decide if this interacts with publish_via_parent. Probably not
+ * (see above).
*/
if (!pubviaroot &&
pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index d097da3c78..43ae131021 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -357,7 +357,8 @@ static bool MergeCheckConstraint(List *constraints, char *name, Node *expr);
static void MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel);
static void MergeConstraintsIntoExisting(Relation child_rel, Relation parent_rel);
static void StoreCatalogInheritance(Oid relationId, List *supers,
- bool child_is_partition);
+ bool child_is_partition,
+ bool publish_via_parent);
static void StoreCatalogInheritance1(Oid relationId, Oid parentOid,
int32 seqNumber, Relation inhRelation,
bool child_is_partition);
@@ -586,6 +587,7 @@ static void ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel,
const char *tablespacename, LOCKMODE lockmode);
static void ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode);
static void ATExecSetTableSpaceNoStorage(Relation rel, Oid newTableSpace);
+static void change_publish_via_parent(Relation rel, bool set);
static void ATExecSetRelOptions(Relation rel, List *defList,
AlterTableType operation,
LOCKMODE lockmode);
@@ -1123,7 +1125,8 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
}
/* Store inheritance information for new rel. */
- StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL);
+ StoreCatalogInheritance(relationId, inheritOids, stmt->partbound != NULL,
+ RelationIsPublishedViaParent(rel));
/*
* Process the partitioning specification (if any) and store the partition
@@ -3325,7 +3328,7 @@ MergeCheckConstraint(List *constraints, char *name, Node *expr)
*/
static void
StoreCatalogInheritance(Oid relationId, List *supers,
- bool child_is_partition)
+ bool child_is_partition, bool publish_via_parent)
{
Relation relation;
int32 seqNumber;
@@ -3336,6 +3339,22 @@ StoreCatalogInheritance(Oid relationId, List *supers,
*/
Assert(OidIsValid(relationId));
+ /*
+ * publish_via_parent requires exactly one parent. Try to keep this
+ * messaging consistent with ALTER TABLE/change_publish_via_parent().
+ *
+ * Earlier, in MergeAttributes, we already checked that we had ownership of
+ * the parent table, so we don't need to check that again the way that
+ * change_publish_via_parent() does.
+ */
+ if (publish_via_parent && list_length(supers) != 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg((list_length(supers) == 0)
+ ? "table \"%s\" does not inherit from any tables"
+ : "table \"%s\" inherits from multiple tables",
+ get_rel_name(relationId))));
+
if (supers == NIL)
return;
@@ -3350,7 +3369,12 @@ StoreCatalogInheritance(Oid relationId, List *supers,
*/
relation = table_open(InheritsRelationId, RowExclusiveLock);
- seqNumber = 1;
+ /*
+ * Normally inhseqno starts at one, but for publish_via_parent, it's given a
+ * sentinel value of zero. (In that case, we'll only go through this loop
+ * once, as the list_length() check above enforces.)
+ */
+ seqNumber = publish_via_parent ? 0 : 1;
foreach(entry, supers)
{
Oid parentOid = lfirst_oid(entry);
@@ -14829,6 +14853,88 @@ ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel, const char *tablespacen
tab->newTableSpace = tablespaceId;
}
+static void
+change_publish_via_parent(Relation rel, bool set)
+{
+ Relation inhRel;
+ ScanKeyData key;
+ SysScanDesc scan;
+ Oid parentOid;
+ Relation parentRel;
+ HeapTuple tuple, copyTuple;
+ Form_pg_inherits form;
+
+ /* Open pg_inherits with RowExclusiveLock so that we can update it. */
+ inhRel = table_open(InheritsRelationId, RowExclusiveLock);
+
+ /*
+ * We have to make sure that an inheritance relationship already exists, and
+ * that there is only one candidate parent for this table.
+ *
+ * TODO: do we have to lock the tables themselves to avoid races?
+ */
+ ScanKeyInit(&key,
+ Anum_pg_inherits_inhrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationGetRelid(rel)));
+
+ scan = systable_beginscan(inhRel, InheritsRelidSeqnoIndexId, true,
+ NULL, 1, &key);
+ tuple = systable_getnext(scan);
+
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" does not inherit from any tables",
+ RelationGetRelationName(rel))));
+
+ form = (Form_pg_inherits) GETSTRUCT(tuple);
+ parentOid = form->inhparent;
+ copyTuple = heap_copytuple(tuple);
+
+ if (HeapTupleIsValid(systable_getnext(scan)))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table \"%s\" inherits from multiple tables",
+ RelationGetRelationName(rel))));
+
+ systable_endscan(scan);
+
+ if (set)
+ {
+ /*
+ * Open our parent table and check permissions. Like ATTACH PARTITION,
+ * we require ownership of the parent table in addition to the child
+ * (which has already been checked in ATPrepCmd).
+ *
+ * TODO: what lock strength is needed here? Is nesting it inside the
+ * inhRel lock a risk?
+ */
+ parentRel = table_open(parentOid, AccessExclusiveLock);
+ ATSimplePermissions(AT_SetRelOptions, parentRel, ATT_TABLE);
+ table_close(parentRel, NoLock); /* TODO: do we need to hang onto the lock? */
+ }
+ else
+ {
+ /*
+ * Ownership of the parent isn't needed for unsetting the flag, just
+ * like we wouldn't check ownership during NO INHERIT.
+ */
+ }
+
+ /*
+ * Mark the inheritance as a logical root by setting inhseqno to zero.
+ * Unmark it by setting inhseqno to one (since there's only one parent).
+ */
+ form = (Form_pg_inherits) GETSTRUCT(copyTuple);
+ form->inhseqno = set ? 0 : 1;
+
+ CatalogTupleUpdate(inhRel, ©Tuple->t_self, copyTuple);
+
+ heap_freetuple(copyTuple);
+ table_close(inhRel, RowExclusiveLock);
+}
+
/*
* Set, reset, or replace reloptions.
*/
@@ -14940,6 +15046,39 @@ ATExecSetRelOptions(Relation rel, List *defList, AlterTableType operation,
}
}
+ /* Extra work for publish_via_parent */
+ if (rel->rd_rel->relkind == RELKIND_RELATION)
+ {
+ List *heap_options = untransformRelOptions(newOptions);
+ ListCell *cell;
+ bool curSetting = RelationIsPublishedViaParent(rel);
+ DefElem *pubElem = NULL;
+
+ foreach (cell, heap_options)
+ {
+ DefElem *defel = lfirst_node(DefElem, cell);
+
+ if (strcmp(defel->defname, "publish_via_parent") == 0)
+ {
+ pubElem = defel;
+ break;
+ }
+ }
+
+ if (pubElem)
+ {
+ bool newSetting = defGetBoolean(pubElem);
+
+ if (newSetting != curSetting)
+ change_publish_via_parent(rel, newSetting);
+ }
+ else if (curSetting)
+ {
+ /* publish_via_parent was RESET */
+ change_publish_via_parent(rel, false);
+ }
+ }
+
/*
* All we need do here is update the pg_class row; the new options will be
* propagated into relcaches during post-commit cache inval.
@@ -15538,6 +15677,13 @@ ATExecAddInherit(Relation child_rel, RangeVar *parent, LOCKMODE lockmode)
trigger_name, RelationGetRelationName(child_rel)),
errdetail("ROW triggers with transition tables are not supported in inheritance hierarchies.")));
+ /* publish_via_parent tables are not allowed to add additional parents. */
+ if (RelationIsPublishedViaParent(child_rel))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
+ "publish_via_parent", RelationGetRelationName(child_rel))));
+
/* OK to create inheritance */
CreateInheritance(child_rel, parent_rel);
@@ -16001,6 +16147,13 @@ ATExecDropInherit(Relation rel, RangeVar *parent, LOCKMODE lockmode)
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot change inheritance of a partition")));
+ /* publish_via_parent tables are not allowed to remove their parent. */
+ if (RelationIsPublishedViaParent(rel))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("relation option \"%s\" prevents table \"%s\" from changing inheritance",
+ "publish_via_parent", RelationGetRelationName(rel))));
+
/*
* AccessShareLock on the parent is probably enough, seeing that DROP
* TABLE doesn't lock parent tables at all. We need some lock since we'll
diff --git a/src/backend/partitioning/partdesc.c b/src/backend/partitioning/partdesc.c
index 65f3d5a5e6..0ed7cbbbc5 100644
--- a/src/backend/partitioning/partdesc.c
+++ b/src/backend/partitioning/partdesc.c
@@ -162,7 +162,8 @@ RelationBuildPartitionDesc(Relation rel, bool omit_detached)
inhoids = find_inheritance_children_extended(RelationGetRelid(rel),
omit_detached, NoLock,
&detached_exist,
- &detached_xmin);
+ &detached_xmin,
+ false);
nparts = list_length(inhoids);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e2cee92cf2..1e6dc70e32 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -771,11 +771,12 @@ copy_read_data(void *outbuf, int minread, int maxread)
/*
* Get information about remote relation in similar fashion the RELATION
* message provides during replication. This function also returns the relation
- * qualifications to be used in the COPY command.
+ * qualifications to be used in the COPY command, and the list of tables to COPY
+ * (which for most tables will contain only one entry).
*/
static void
fetch_remote_table_info(char *nspname, char *relname,
- LogicalRepRelation *lrel, List **qual)
+ LogicalRepRelation *lrel, List **qual, List **to_copy)
{
WalRcvExecResult *res;
StringInfoData cmd;
@@ -1086,6 +1087,87 @@ fetch_remote_table_info(char *nspname, char *relname,
walrcv_clear_result(res);
}
+ /*
+ * See if there are any other tables to be copied besides the original. This
+ * happens when a descendant in the inheritance relationship is marked with
+ * publish_via_parent and is part of a publication with
+ * publish_via_partition_root = true.
+ *
+ * FIXME: if two publications have conflicting settings for
+ * publish_via_partition_root, this behaves strangely. It looks like that's
+ * true for regular partitions too...
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000)
+ {
+ Oid descRow[] = {TEXTOID, TEXTOID};
+ StringInfoData pub_names;
+
+ /* Build the pubname list. */
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ char *pubname = strVal(lfirst(lc));
+
+ if (foreach_current_index(lc) > 0)
+ appendStringInfoString(&pub_names, ", ");
+
+ appendStringInfoString(&pub_names, quote_literal_cstr(pubname));
+ }
+
+ /*
+ * Ask the server what it wants us to sync.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT n.nspname, c.relname"
+ " FROM pg_catalog.pg_publication p,"
+ " pg_catalog.pg_class c"
+ " JOIN pg_catalog.pg_namespace n"
+ " ON (c.relnamespace = n.oid)"
+ " JOIN LATERAL pg_catalog.pg_get_publication_rels_to_sync(%u::regclass, p.pubname) relid"
+ " ON (c.oid = relid)"
+ " WHERE p.pubname IN ( %s )",
+ lrel->remoteid, pub_names.data);
+
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(descRow), descRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch logical descendants for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *desc_nspname;
+ char *desc_relname;
+ char *quoted;
+
+ desc_nspname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+ desc_relname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ quoted = quote_qualified_identifier(desc_nspname, desc_relname);
+ *to_copy = lappend(*to_copy, quoted);
+
+ ExecClearTuple(slot);
+ }
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+
+ pfree(pub_names.data);
+ }
+ else
+ {
+ /* For older servers, we only COPY the table itself. */
+ char *quoted = quote_qualified_identifier(lrel->nspname,
+ lrel->relname);
+ *to_copy = lappend(*to_copy, quoted);
+ }
+
pfree(cmd.data);
}
@@ -1100,6 +1182,8 @@ copy_table(Relation rel)
LogicalRepRelMapEntry *relmapentry;
LogicalRepRelation lrel;
List *qual = NIL;
+ List *to_copy = NIL;
+ ListCell *cur;
WalRcvExecResult *res;
StringInfoData cmd;
CopyFromState cstate;
@@ -1109,7 +1193,8 @@ copy_table(Relation rel)
/* Get the publisher relation info. */
fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
- RelationGetRelationName(rel), &lrel, &qual);
+ RelationGetRelationName(rel), &lrel, &qual,
+ &to_copy);
/* Put the relation into relmap. */
logicalrep_relmap_update(&lrel);
@@ -1118,105 +1203,109 @@ copy_table(Relation rel)
relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
Assert(rel == relmapentry->localrel);
- /* Start copy on the publisher. */
- initStringInfo(&cmd);
-
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ foreach(cur, to_copy)
{
- appendStringInfo(&cmd, "COPY %s (",
- quote_qualified_identifier(lrel.nspname, lrel.relname));
-
- /*
- * XXX Do we need to list the columns in all cases? Maybe we're
- * replicating all columns?
- */
- for (int i = 0; i < lrel.natts; i++)
- {
- if (i > 0)
- appendStringInfoString(&cmd, ", ");
+ char *quoted_name = lfirst(cur);
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- }
+ /* Start copy on the publisher. */
+ initStringInfo(&cmd);
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
- else
- {
- /*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
- */
- appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ /* Regular table with no row filter */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
- appendStringInfoString(&cmd, ", ");
- }
+ appendStringInfo(&cmd, "COPY %s (", quoted_name);
- appendStringInfoString(&cmd, " FROM ");
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're
+ * replicating all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
- /*
- * For regular tables, make sure we don't copy data from a child that
- * inherits the named table as those will be copied separately.
- */
- if (lrel.relkind == RELKIND_RELATION)
- appendStringInfoString(&cmd, "ONLY ");
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
- appendStringInfoString(&cmd, quote_qualified_identifier(lrel.nspname, lrel.relname));
- /* list of OR'ed filters */
- if (qual != NIL)
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
+ else
{
- ListCell *lc;
- char *q = strVal(linitial(qual));
+ /*
+ * For non-tables and tables with row filters, we need to do COPY
+ * (SELECT ...), but we can't just do SELECT * because we need to not
+ * copy generated columns. For tables with any row filters, build a
+ * SELECT query with OR'ed row filters for COPY.
+ */
+ appendStringInfoString(&cmd, "COPY (SELECT ");
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+
+ appendStringInfoString(&cmd, " FROM ");
+
+ /*
+ * For regular tables, make sure we don't copy data from a child that
+ * inherits the named table as those will be copied separately.
+ */
+ if (lrel.relkind == RELKIND_RELATION)
+ appendStringInfoString(&cmd, "ONLY ");
- appendStringInfo(&cmd, " WHERE %s", q);
- for_each_from(lc, qual, 1)
+ appendStringInfoString(&cmd, quoted_name);
+ /* list of OR'ed filters */
+ if (qual != NIL)
{
- q = strVal(lfirst(lc));
- appendStringInfo(&cmd, " OR %s", q);
+ ListCell *lc;
+ char *q = strVal(linitial(qual));
+
+ appendStringInfo(&cmd, " WHERE %s", q);
+ for_each_from(lc, qual, 1)
+ {
+ q = strVal(lfirst(lc));
+ appendStringInfo(&cmd, " OR %s", q);
+ }
+ list_free_deep(qual);
}
- list_free_deep(qual);
- }
- appendStringInfoString(&cmd, ") TO STDOUT");
- }
+ appendStringInfoString(&cmd, ") TO STDOUT");
+ }
- /*
- * Prior to v16, initial table synchronization will use text format even
- * if the binary option is enabled for a subscription.
- */
- if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000 &&
- MySubscription->binary)
- {
- appendStringInfoString(&cmd, " WITH (FORMAT binary)");
- options = list_make1(makeDefElem("format",
- (Node *) makeString("binary"), -1));
- }
+ /*
+ * Prior to v16, initial table synchronization will use text format even
+ * if the binary option is enabled for a subscription.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 160000 &&
+ MySubscription->binary)
+ {
+ appendStringInfoString(&cmd, " WITH (FORMAT binary)");
+ options = list_make1(makeDefElem("format",
+ (Node *) makeString("binary"), -1));
+ }
- res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
- pfree(cmd.data);
- if (res->status != WALRCV_OK_COPY_OUT)
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("could not start initial contents copy for table \"%s.%s\": %s",
- lrel.nspname, lrel.relname, res->err)));
- walrcv_clear_result(res);
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
+ pfree(cmd.data);
+ if (res->status != WALRCV_OK_COPY_OUT)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not start initial contents copy for table \"%s.%s\" from remote %s: %s",
+ lrel.nspname, lrel.relname, quoted_name, res->err)));
+ walrcv_clear_result(res);
- copybuf = makeStringInfo();
+ copybuf = makeStringInfo();
- pstate = make_parsestate(NULL);
- (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
- NULL, false, false);
+ pstate = make_parsestate(NULL);
+ (void) addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
+ NULL, false, false);
- attnamelist = make_copy_attnamelist(relmapentry);
- cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
+ attnamelist = make_copy_attnamelist(relmapentry);
+ cstate = BeginCopyFrom(pstate, rel, NULL, NULL, false, copy_read_data, attnamelist, options);
- /* Do the copy */
- (void) CopyFrom(cstate);
+ /* Do the copy */
+ (void) CopyFrom(cstate);
+ }
logicalrep_rel_close(relmapentry, NoLock);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 640676e7dc..7c54b46ac7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1462,7 +1462,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Switch relation if publishing via root. */
if (relentry->publish_as_relid != RelationGetRelid(relation))
{
- Assert(relation->rd_rel->relispartition);
ancestor = RelationIdGetRelation(relentry->publish_as_relid);
targetrel = ancestor;
}
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 0758fe5ea0..334253e489 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -4600,6 +4600,30 @@ my %tests = (
no_table_access_method => 1,
only_dump_measurement => 1,
},
+ },
+
+ 'CREATE TABLE regress_pg_dump_publish_via_parent' => {
+ create_order => 3,
+ create_sql => '
+ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent (col1 int);
+ CREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1() INHERITS (dump_test.regress_pg_dump_publish_via_parent) WITH (publish_via_parent);',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent (\E
+ \n\s+\Qcol1 integer\E
+ \n\);
+ (\n.*)+?
+ \n\QCREATE TABLE dump_test.regress_pg_dump_publish_via_parent_1 (\E
+ \n\)
+ \n\QINHERITS (dump_test.regress_pg_dump_publish_via_parent)\E
+ \n\QWITH (publish_via_parent='true')\E;/xm,
+ like => {
+ %full_runs, %dump_test_schema_runs, section_pre_data => 1,
+ },
+ unlike => {
+ binary_upgrade => 1,
+ exclude_dump_test_schema => 1,
+ only_dump_measurement => 1,
+ },
});
#########################################
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 779fdc90cb..000a996e69 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1306,6 +1306,7 @@ static const char *const table_storage_parameters[] = {
"fillfactor",
"log_autovacuum_min_duration",
"parallel_workers",
+ "publish_via_parent",
"toast.autovacuum_enabled",
"toast.autovacuum_freeze_max_age",
"toast.autovacuum_freeze_min_age",
diff --git a/src/include/catalog/pg_inherits.h b/src/include/catalog/pg_inherits.h
index ce154ab943..fc539e7401 100644
--- a/src/include/catalog/pg_inherits.h
+++ b/src/include/catalog/pg_inherits.h
@@ -23,6 +23,7 @@
#include "nodes/pg_list.h"
#include "storage/lock.h"
+#include "utils/relcache.h"
/* ----------------
* pg_inherits definition. cpp turns this into
@@ -46,14 +47,17 @@ typedef FormData_pg_inherits *Form_pg_inherits;
DECLARE_UNIQUE_INDEX_PKEY(pg_inherits_relid_seqno_index, 2680, InheritsRelidSeqnoIndexId, on pg_inherits using btree(inhrelid oid_ops, inhseqno int4_ops));
DECLARE_INDEX(pg_inherits_parent_index, 2187, InheritsParentIndexId, on pg_inherits using btree(inhparent oid_ops));
+DECLARE_INDEX(pg_inherits_parent_seqno_index, 8138, InheritsParentSeqnoIndexId, on pg_inherits using btree(inhparent oid_ops, inhseqno int4_ops));
extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);
extern List *find_inheritance_children_extended(Oid parentrelId, bool omit_detached,
- LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin);
+ LOCKMODE lockmode, bool *detached_exist, TransactionId *detached_xmin,
+ bool logical_only);
extern List *find_all_inheritors(Oid parentrelId, LOCKMODE lockmode,
List **numparents);
+extern List *find_all_logical_inheritors(Oid parentrelId);
extern bool has_subclass(Oid relationId);
extern bool has_superclass(Oid relationId);
extern bool typeInheritsFrom(Oid subclassTypeId, Oid superclassTypeId);
@@ -63,5 +67,7 @@ extern bool DeleteInheritsTuple(Oid inhrelid, Oid inhparent,
bool expect_detach_pending,
const char *childname);
extern bool PartitionHasPendingDetach(Oid partoid);
+extern List *get_logical_ancestors(Oid relid, bool is_partition);
+extern bool has_logical_parent(Relation inhRel, Oid relid);
#endif /* PG_INHERITS_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0598af0cf9..b0456ac0f2 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11852,6 +11852,13 @@
proname => 'pg_relation_is_publishable', provolatile => 's',
prorettype => 'bool', proargtypes => 'regclass',
prosrc => 'pg_relation_is_publishable' },
+{ oid => '8137',
+ descr => 'get the relations to copy from all specified publications during a table\'s initial sync',
+ proname => 'pg_get_publication_rels_to_sync', prorows => '10',
+ provariadic => 'text', proretset => 't', provolatile => 's',
+ prorettype => 'regclass', proargtypes => 'regclass _text',
+ proargmodes => '{i,v}', proargnames => '{rootid,pubnames}',
+ prosrc => 'pg_get_publication_rels_to_sync' },
{ oid => '8139',
descr => 'get information on how a relation will be published via a list of publications',
proname => 'pg_get_relation_publishing_info', provariadic => 'text',
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 1426a353cd..3c5ea5a37a 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -339,6 +339,7 @@ typedef struct StdRdOptions
int toast_tuple_target; /* target for tuple toasting */
AutoVacOpts autovacuum; /* autovacuum-related options */
bool user_catalog_table; /* use as an additional catalog relation */
+ bool publish_via_parent; /* publish via parent's relid for logrep */
int parallel_workers; /* max number of parallel workers */
StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */
bool vacuum_truncate; /* enables vacuum to truncate a relation */
@@ -388,6 +389,17 @@ typedef struct StdRdOptions
(relation)->rd_rel->relkind == RELKIND_MATVIEW) ? \
((StdRdOptions *) (relation)->rd_options)->user_catalog_table : false)
+/*
+ * RelationIsPublishedViaParent
+ * Returns whether the relation's contents should be logically replicated
+ * via its parent table's relid, when published under the effects of
+ * publish_via_partition_root. Note multiple eval of argument!
+ */
+#define RelationIsPublishedViaParent(relation) \
+ ((relation)->rd_options && \
+ ((relation)->rd_rel->relkind == RELKIND_RELATION) && \
+ ((StdRdOptions *) (relation)->rd_options)->publish_via_parent)
+
/*
* RelationGetParallelWorkers
* Returns the relation's parallel_workers reloption setting.
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 85c217dadb..65e5cf6893 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -5,6 +5,17 @@ CREATE ROLE regress_publication_user LOGIN SUPERUSER;
CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_sync (VARIADIC pubnames text[])
+ RETURNS TABLE (published regclass, synced regclass) AS $$
+ -- Show each published table alongside the tables to be copied into it during
+ -- initial sync.
+ SELECT t.published, synced
+ FROM (SELECT DISTINCT relid::regclass AS published
+ FROM pg_get_publication_tables(VARIADIC pubnames)) t
+ JOIN LATERAL pg_get_publication_rels_to_sync(t.published, VARIADIC pubnames) synced
+ ON true
+ ORDER BY 1, 2;
+$$ LANGUAGE sql;
CREATE FUNCTION published_stream (VARIADIC pubnames text[])
RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
-- For each publication, show each published root alongside the tables which
@@ -1703,6 +1714,12 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+SELECT * FROM published_sync('pub');
+ published | synced
+-----------+-----------
+ sch1.tbl1 | sch1.tbl1
+(1 row)
+
SELECT * FROM published_stream('pub');
pubname | published | synced
---------+-----------+-----------------
@@ -1736,6 +1753,12 @@ SELECT * FROM pg_publication_tables;
pub | sch2 | tbl1_part1 | {a} |
(1 row)
+SELECT * FROM published_sync('pub');
+ published | synced
+-----------------+-----------------
+ sch2.tbl1_part1 | sch2.tbl1_part1
+(1 row)
+
SELECT * FROM published_stream('pub');
pubname | published | synced
---------+-----------------+-----------------
@@ -1758,12 +1781,348 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1 | {a} |
(1 row)
+-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+-- should do nothing at all
+ALTER TABLE sch1.iroot SET (publish_via_parent = false);
+ALTER TABLE sch1.iroot RESET (publish_via_parent);
+-- marking roots between unrelated tables is not allowed
+ALTER TABLE sch1.ipart1 SET (publish_via_parent);
+ERROR: table "ipart1" does not inherit from any tables
+CREATE TABLE fail (a int) WITH (publish_via_parent);
+ERROR: table "fail" does not inherit from any tables
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
+ SET (publish_via_parent);
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
+ SET (publish_via_parent);
+ERROR: table "ipart2" inherits from multiple tables
+-- once publish_via_parent is set, inheritance cannot be changed
+ALTER TABLE sch1.ipart2 SET (publish_via_parent);
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
+ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+ERROR: relation option "publish_via_parent" prevents table "ipart2" from changing inheritance
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should fail
+ERROR: must be owner of table part
+RESET SESSION AUTHORIZATION;
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should also fail
+ERROR: must be owner of table root
+CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+ERROR: must be owner of table root
+RESET SESSION AUTHORIZATION;
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+CREATE PUBLICATION ipub_all FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_one FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
+CREATE PUBLICATION ipub_two FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
+-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
+-- ------------------ --------------------- ---------------------
+-- iroot
+-- +- ipart1
+-- | +- ipart1_a - ipart1_a - ipart1_a
+-- | +- ipart1_a1
+-- +- ipart2
+-- A subscription to only ipub_all should see every individual table.
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_all');
+ relid
+----------------
+ sch1.iroot
+ sch1.ipart1
+ sch1.ipart2
+ sch1.ipart1_a
+ sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_sync('ipub_all');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a | sch1.ipart1_a
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_all');
+ pubname | published | synced
+----------+----------------+----------------
+ ipub_all | sch1.iroot | sch1.iroot
+ ipub_all | sch1.ipart1 | sch1.ipart1
+ ipub_all | sch1.ipart2 | sch1.ipart2
+ ipub_all | sch1.ipart1_a | sch1.ipart1_a
+ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+-- A subscription to both ipub_all and ipub_one shouldn't change the initial
+-- sync, since there is no alternative root being published for ipart1_a. It
+-- will be duplicated in the stream list, since TODO
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+----------+----------------
+ ipub_all | sch1.iroot
+ ipub_all | sch1.ipart1
+ ipub_all | sch1.ipart2
+ ipub_all | sch1.ipart1_a
+ ipub_all | sch1.ipart1_a1
+ ipub_one | sch1.ipart1_a
+(6 rows)
+
+SELECT * FROM published_sync('ipub_all', 'ipub_one');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a | sch1.ipart1_a
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_all', 'ipub_one');
+ pubname | published | synced
+----------+----------------+----------------
+ ipub_all | sch1.iroot | sch1.iroot
+ ipub_all | sch1.ipart1 | sch1.ipart1
+ ipub_all | sch1.ipart2 | sch1.ipart2
+ ipub_all | sch1.ipart1_a | sch1.ipart1_a
+ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+ ipub_one | sch1.ipart1_a | sch1.ipart1_a
+(6 rows)
+
+ALTER PUBLICATION ipub_one ADD TABLE ONLY sch1.ipart1;
+-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
+-- ------------------ --------------------- ---------------------
+-- iroot
+-- +- ipart1 - ipart1
+-- | +- ipart1_a +- ipart1_a - ipart1_a
+-- | +- ipart1_a1
+-- +- ipart2
+-- Adding ipart1_a's parent table to ipub_one results in ipart1_a1 being
+-- "stolen" from the ipub_all stream, to prevent its data from being duplicated.
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+----------+----------------
+ ipub_all | sch1.iroot
+ ipub_all | sch1.ipart1
+ ipub_all | sch1.ipart2
+ ipub_all | sch1.ipart1_a1
+ ipub_one | sch1.ipart1
+(5 rows)
+
+SELECT * FROM published_sync('ipub_all', 'ipub_one');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_all', 'ipub_one');
+ pubname | published | synced
+----------+----------------+----------------
+ ipub_all | sch1.iroot | sch1.iroot
+ ipub_all | sch1.ipart1 | sch1.ipart1
+ ipub_all | sch1.ipart1 | sch1.ipart1_a
+ ipub_all | sch1.ipart2 | sch1.ipart2
+ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+ ipub_one | sch1.ipart1 | sch1.ipart1
+ ipub_one | sch1.ipart1 | sch1.ipart1_a
+(7 rows)
+
+-- Including ipub_two in the subscription list should change nothing about the
+-- sync. ipub_two will gain an entry for ipart1_a in the streaming list.
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+----------+----------------
+ ipub_all | sch1.iroot
+ ipub_all | sch1.ipart1
+ ipub_all | sch1.ipart2
+ ipub_all | sch1.ipart1_a1
+ ipub_one | sch1.ipart1
+(5 rows)
+
+SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1 | sch1.ipart1_a
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+ pubname | published | synced
+----------+----------------+----------------
+ ipub_all | sch1.iroot | sch1.iroot
+ ipub_all | sch1.ipart1 | sch1.ipart1
+ ipub_all | sch1.ipart1 | sch1.ipart1_a
+ ipub_all | sch1.ipart2 | sch1.ipart2
+ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+ ipub_one | sch1.ipart1 | sch1.ipart1
+ ipub_one | sch1.ipart1 | sch1.ipart1_a
+ ipub_two | sch1.ipart1 | sch1.ipart1_a
+(8 rows)
+
+ALTER PUBLICATION ipub_two ADD TABLE ONLY sch1.iroot;
+-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
+-- ------------------ --------------------- ---------------------
+-- iroot iroot
+-- +- ipart1 - ipart1 |
+-- | +- ipart1_a +- ipart1_a +---- ipart1_a
+-- | +- ipart1_a1
+-- +- ipart2
+-- Adding iroot to ipub_two ends up stealing both ipart1 and ipart1_a from the
+-- other publications, again so that no data is double-published.
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+----------+----------------
+ ipub_all | sch1.iroot
+ ipub_all | sch1.ipart2
+ ipub_all | sch1.ipart1_a1
+ ipub_two | sch1.iroot
+(4 rows)
+
+SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.iroot | sch1.ipart1
+ sch1.iroot | sch1.ipart1_a
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+ pubname | published | synced
+----------+----------------+----------------
+ ipub_all | sch1.iroot | sch1.iroot
+ ipub_all | sch1.iroot | sch1.ipart1
+ ipub_all | sch1.iroot | sch1.ipart1_a
+ ipub_all | sch1.ipart2 | sch1.ipart2
+ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+ ipub_one | sch1.iroot | sch1.ipart1
+ ipub_one | sch1.iroot | sch1.ipart1_a
+ ipub_two | sch1.iroot | sch1.iroot
+ ipub_two | sch1.iroot | sch1.ipart1_a
+(9 rows)
+
+-- "Detaching" partitions should change the subscriptions
+ALTER TABLE sch1.ipart1_a SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+----------+----------------
+ ipub_all | sch1.iroot
+ ipub_all | sch1.ipart2
+ ipub_all | sch1.ipart1_a
+ ipub_all | sch1.ipart1_a1
+ ipub_one | sch1.ipart1_a
+ ipub_two | sch1.iroot
+ ipub_two | sch1.ipart1_a
+(7 rows)
+
+SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+ published | synced
+----------------+----------------
+ sch1.iroot | sch1.iroot
+ sch1.iroot | sch1.ipart1
+ sch1.ipart2 | sch1.ipart2
+ sch1.ipart1_a | sch1.ipart1_a
+ sch1.ipart1_a1 | sch1.ipart1_a1
+(5 rows)
+
+SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+ pubname | published | synced
+----------+----------------+----------------
+ ipub_all | sch1.iroot | sch1.iroot
+ ipub_all | sch1.iroot | sch1.ipart1
+ ipub_all | sch1.ipart2 | sch1.ipart2
+ ipub_all | sch1.ipart1_a | sch1.ipart1_a
+ ipub_all | sch1.ipart1_a1 | sch1.ipart1_a1
+ ipub_one | sch1.iroot | sch1.ipart1
+ ipub_one | sch1.ipart1_a | sch1.ipart1_a
+ ipub_two | sch1.iroot | sch1.iroot
+ ipub_two | sch1.ipart1_a | sch1.ipart1_a
+(9 rows)
+
+ALTER TABLE sch1.ipart1 RESET (publish_via_parent);
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+ pubname | relid
+----------+---------------
+ ipub_one | sch1.ipart1
+ ipub_one | sch1.ipart1_a
+ ipub_two | sch1.iroot
+ ipub_two | sch1.ipart1_a
+(4 rows)
+
+SELECT * FROM published_sync('ipub_one', 'ipub_two');
+ published | synced
+---------------+---------------
+ sch1.iroot | sch1.iroot
+ sch1.ipart1 | sch1.ipart1
+ sch1.ipart1_a | sch1.ipart1_a
+(3 rows)
+
+SELECT * FROM published_stream('ipub_one', 'ipub_two');
+ pubname | published | synced
+----------+---------------+---------------
+ ipub_one | sch1.ipart1 | sch1.ipart1
+ ipub_one | sch1.ipart1_a | sch1.ipart1_a
+ ipub_two | sch1.iroot | sch1.iroot
+ ipub_two | sch1.ipart1_a | sch1.ipart1_a
+(4 rows)
+
+DROP PUBLICATION ipub_all;
+DROP PUBLICATION ipub_one;
+DROP PUBLICATION ipub_two;
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1_a1;
+DROP TABLE sch1.ipart1_a;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
DROP FUNCTION published_stream;
+DROP FUNCTION published_sync;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
DROP ROLE regress_publication_user_dummy;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 6bf9554d48..43320dfa92 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -6,6 +6,18 @@ CREATE ROLE regress_publication_user2;
CREATE ROLE regress_publication_user_dummy LOGIN NOSUPERUSER;
SET SESSION AUTHORIZATION 'regress_publication_user';
+CREATE FUNCTION published_sync (VARIADIC pubnames text[])
+ RETURNS TABLE (published regclass, synced regclass) AS $$
+ -- Show each published table alongside the tables to be copied into it during
+ -- initial sync.
+ SELECT t.published, synced
+ FROM (SELECT DISTINCT relid::regclass AS published
+ FROM pg_get_publication_tables(VARIADIC pubnames)) t
+ JOIN LATERAL pg_get_publication_rels_to_sync(t.published, VARIADIC pubnames) synced
+ ON true
+ ORDER BY 1, 2;
+$$ LANGUAGE sql;
+
CREATE FUNCTION published_stream (VARIADIC pubnames text[])
RETURNS TABLE (pubname text, published regclass, synced regclass) AS $$
-- For each publication, show each published root alongside the tables which
@@ -1076,6 +1088,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_sync('pub');
SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
@@ -1091,6 +1104,7 @@ SELECT * FROM pg_publication_tables;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM published_sync('pub');
SELECT * FROM published_stream('pub');
DROP PUBLICATION pub;
@@ -1105,12 +1119,164 @@ ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (
CREATE PUBLICATION pub FOR TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
+-- Sanity check cases for publish_via_parent.
+CREATE TABLE sch1.iroot (a int);
+CREATE TABLE sch1.ipart1 (a int);
+CREATE TABLE sch1.ipart2 () INHERITS (sch1.iroot);
+
+-- should do nothing at all
+ALTER TABLE sch1.iroot SET (publish_via_parent = false);
+ALTER TABLE sch1.iroot RESET (publish_via_parent);
+
+-- marking roots between unrelated tables is not allowed
+ALTER TABLE sch1.ipart1 SET (publish_via_parent);
+CREATE TABLE fail (a int) WITH (publish_via_parent);
+
+-- establishing an inheritance relationship fixes the problem
+ALTER TABLE sch1.ipart1 INHERIT sch1.iroot,
+ SET (publish_via_parent);
+
+-- but multiple inheritance is not allowed
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1,
+ SET (publish_via_parent);
+
+-- once publish_via_parent is set, inheritance cannot be changed
+ALTER TABLE sch1.ipart2 SET (publish_via_parent);
+ALTER TABLE sch1.ipart2 NO INHERIT sch1.iroot;
+ALTER TABLE sch1.ipart2 INHERIT sch1.ipart1;
+
+-- table ownership must match, like ATTACH PARTITION
+CREATE ROLE regress_test_me;
+CREATE ROLE regress_test_not_me;
+CREATE TABLE root (a int);
+CREATE TABLE part () INHERITS (root);
+
+ALTER TABLE root OWNER TO regress_test_me;
+ALTER TABLE part OWNER TO regress_test_not_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should fail
+RESET SESSION AUTHORIZATION;
+
+ALTER TABLE root OWNER TO regress_test_not_me;
+ALTER TABLE part OWNER TO regress_test_me;
+SET SESSION AUTHORIZATION regress_test_me;
+ALTER TABLE part SET (publish_via_parent); -- should also fail
+CREATE TABLE fail () INHERITS (root) WITH (publish_via_parent);
+RESET SESSION AUTHORIZATION;
+
+DROP TABLE root, part;
+DROP ROLE regress_test_not_me;
+DROP ROLE regress_test_me;
+
+-- Mixed publication settings for publish_via_partition_root, at different
+-- levels of the inheritance tree, to pin correct behavior in the worst cases.
+CREATE TABLE sch1.ipart1_a () INHERITS (sch1.ipart1) WITH (publish_via_parent);
+CREATE TABLE sch1.ipart1_a1 () INHERITS (sch1.ipart1_a);
+ALTER TABLE sch1.ipart1_a1 SET (publish_via_parent);
+
+CREATE PUBLICATION ipub_all FOR TABLE sch1.iroot;
+CREATE PUBLICATION ipub_one FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
+CREATE PUBLICATION ipub_two FOR TABLE ONLY sch1.ipart1_a
+ WITH (publish_via_partition_root);
+
+-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
+-- ------------------ --------------------- ---------------------
+-- iroot
+-- +- ipart1
+-- | +- ipart1_a - ipart1_a - ipart1_a
+-- | +- ipart1_a1
+-- +- ipart2
+
+-- A subscription to only ipub_all should see every individual table.
+SELECT relid::regclass FROM pg_get_publication_tables('ipub_all');
+SELECT * FROM published_sync('ipub_all');
+SELECT * FROM published_stream('ipub_all');
+
+-- A subscription to both ipub_all and ipub_one shouldn't change the initial
+-- sync, since there is no alternative root being published for ipart1_a. It
+-- will be duplicated in the stream list, since TODO
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_all', 'ipub_one');
+SELECT * FROM published_stream('ipub_all', 'ipub_one');
+
+ALTER PUBLICATION ipub_one ADD TABLE ONLY sch1.ipart1;
+
+-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
+-- ------------------ --------------------- ---------------------
+-- iroot
+-- +- ipart1 - ipart1
+-- | +- ipart1_a +- ipart1_a - ipart1_a
+-- | +- ipart1_a1
+-- +- ipart2
+
+-- Adding ipart1_a's parent table to ipub_one results in ipart1_a1 being
+-- "stolen" from the ipub_all stream, to prevent its data from being duplicated.
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_all', 'ipub_one');
+SELECT * FROM published_stream('ipub_all', 'ipub_one');
+
+-- Including ipub_two in the subscription list should change nothing about the
+-- sync. ipub_two will gain an entry for ipart1_a in the streaming list.
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+
+ALTER PUBLICATION ipub_two ADD TABLE ONLY sch1.iroot;
+
+-- ipub_all ipub_one (pubviaroot) ipub_two (pubviaroot)
+-- ------------------ --------------------- ---------------------
+-- iroot iroot
+-- +- ipart1 - ipart1 |
+-- | +- ipart1_a +- ipart1_a +---- ipart1_a
+-- | +- ipart1_a1
+-- +- ipart2
+
+-- Adding iroot to ipub_two ends up stealing both ipart1 and ipart1_a from the
+-- other publications, again so that no data is double-published.
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+
+-- "Detaching" partitions should change the subscriptions
+ALTER TABLE sch1.ipart1_a SET (publish_via_parent = false);
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_all', 'ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_all', 'ipub_one', 'ipub_two');
+SELECT * FROM published_stream('ipub_all', 'ipub_one', 'ipub_two');
+
+ALTER TABLE sch1.ipart1 RESET (publish_via_parent);
+SELECT pubname, relid::regclass
+ FROM pg_get_publication_tables('ipub_one', 'ipub_two') t
+ JOIN pg_publication p ON (p.oid = t.pubid);
+SELECT * FROM published_sync('ipub_one', 'ipub_two');
+SELECT * FROM published_stream('ipub_one', 'ipub_two');
+
+DROP PUBLICATION ipub_all;
+DROP PUBLICATION ipub_one;
+DROP PUBLICATION ipub_two;
+
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP TABLE sch1.ipart1_a1;
+DROP TABLE sch1.ipart1_a;
+DROP TABLE sch1.ipart1;
+DROP TABLE sch1.ipart2;
+DROP TABLE sch1.iroot;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
DROP FUNCTION published_stream;
+DROP FUNCTION published_sync;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_publication_user, regress_publication_user2;
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 275fb3b525..09e3f2d028 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -388,6 +388,59 @@ $node_subscriber1->append_conf('postgresql.conf',
"log_min_messages = warning");
$node_subscriber1->reload;
+# Make sure standard inheritance setups aren't broken by the new
+# publish_via_parent handling.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (LIKE itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (LIKE itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1_2 VALUES (2, 'itab1')");
+
+# Regression: Create a publication for an unrelated table and set
+# publish_via_partition_root. This should have no effect at all, since itab1
+# isn't supposed to be using this publication...
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION dummy_pub FOR TABLE tab1 WITH (publish_via_partition_root = true)");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 ADD PUBLICATION dummy_pub");
+$node_subscriber2->wait_for_subscription_sync;
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1");
+is($result, qq(0|itab1), 'initial data synced for itab1 on subscriber 2');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_1");
+is($result, qq(1|itab1), 'initial data synced for itab1_1 on subscriber 2');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_2");
+is($result, qq(2|itab1), 'initial data synced for itab1_2 on subscriber 2');
+
+$node_publisher->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+$node_subscriber2->safe_psql('postgres',
+ "DROP TABLE itab1 CASCADE;");
+
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 DROP PUBLICATION dummy_pub");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION dummy_pub");
+
# Tests for replication using root table identity and schema
# publisher
@@ -886,4 +939,263 @@ $result = $node_subscriber2->safe_psql('postgres',
"SELECT a, b, c FROM tab5_1 ORDER BY 1");
is($result, qq(4||1), 'updates of tab5 replicated correctly');
+# Test that replication works for older inheritance/trigger setups as well.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab1_2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab1_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab1_1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO itab1_2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab1_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab1_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab1_2 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (CHECK (a = 1)) INHERITS (itab2)");
+
+$node_publisher->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab2_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO itab2_1 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_publisher->safe_psql('postgres', "
+ CREATE TRIGGER itab2_trigger
+ BEFORE INSERT ON itab2
+ FOR EACH ROW EXECUTE FUNCTION itab2_trigger();");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab2_1 SET (publish_via_parent = true)");
+
+# itab2_1 should be published using its own identity here, since its parent is
+# not included. itab1_1 should be published via its parent, itab1, without
+# duplicating the rows.
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub_viaroot ADD TABLE itab1, itab1_1, itab2_1");
+
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (0, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (2, 'itab1')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (0, 'itab2')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2')");
+
+# Subscriber 1 only subscribes to some of the partitions, and does not set up
+# partition triggers, to check for the correct routing.
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab1_1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber1->safe_psql('postgres',
+ "CREATE TABLE itab2_1 (a int, b text)");
+
+# Subscriber 2 has different partition names for itab1, and it doesn't partition
+# itab2 at all.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part1 (CHECK (a = 1)) INHERITS (itab1)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab1_part2 (CHECK (a = 2)) INHERITS (itab1)");
+
+$node_subscriber2->safe_psql('postgres', "
+ CREATE OR REPLACE FUNCTION itab_trigger()
+ RETURNS TRIGGER AS \$\$
+ BEGIN
+ IF ( NEW.a = 1 ) THEN INSERT INTO public.itab1_part1 VALUES (NEW.*);
+ ELSIF ( NEW.a = 2 ) THEN INSERT INTO public.itab1_part2 VALUES (NEW.*);
+ ELSE RETURN NEW;
+ END IF;
+ RETURN NULL;
+ END;
+ \$\$
+ LANGUAGE plpgsql;");
+$node_subscriber2->safe_psql('postgres', "
+ CREATE TRIGGER itab_trigger
+ BEFORE INSERT ON itab1
+ FOR EACH ROW EXECUTE FUNCTION itab_trigger();");
+$node_subscriber2->safe_psql('postgres', "
+ ALTER TABLE itab1 ENABLE ALWAYS TRIGGER itab_trigger;");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab2 (a int, b text)");
+
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot REFRESH PUBLICATION");
+$node_subscriber2->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub2 REFRESH PUBLICATION");
+
+$node_subscriber1->wait_for_subscription_sync;
+$node_subscriber2->wait_for_subscription_sync;
+
+# check that data is synced correctly
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 1');
+
+# all of the data should have been routed to itab1 directly (there are no
+# triggers on subscriber 1 to move it elsewhere)
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data correctly routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2), 'initial data synced for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+2|itab1), 'initial data synced for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1), 'initial data correctly routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2), 'initial data synced for itab2 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab1 VALUES (1, 'itab1-new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab2 VALUES (1, 'itab2-new')");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+$node_publisher->wait_for_catchup('sub2');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 1');
+
+$result = $node_subscriber1->safe_psql('postgres',
+ "SELECT a, b FROM itab2_1 ORDER BY 1, 2");
+is($result, qq(1|itab2
+1|itab2-new), 'new data routed for itab2_1 on subscriber 1');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1 ORDER BY 1, 2");
+is($result, qq(0|itab1
+1|itab1
+1|itab1-new
+2|itab1), 'new data routed for itab1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab1_part1 ORDER BY 1, 2");
+is($result, qq(1|itab1
+1|itab1-new), 'new data moved to itab1_part1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM itab2 ORDER BY 1, 2");
+is($result, qq(0|itab2
+1|itab2
+1|itab2-new), 'new data routed for itab2 on subscriber 2');
+
+# Finally, check the effect of mixed publish_via_partition_root settings on
+# logical roots.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3 (a int, b text)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3_1 () INHERITS (itab3)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab3_1 SET (publish_via_parent = true)");
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE itab3_1_1 SET (publish_via_parent = true)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3 VALUES (1, 'itab3')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3_1 VALUES (2, 'itab3_1')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO itab3_1_1 VALUES (3, 'itab3_1_1')");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION ipub3 FOR TABLE itab3 *");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION ipub3_1 FOR TABLE itab3_1 * WITH (publish_via_partition_root = true)");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3 (a int, b text)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3_1 () INHERITS (itab3)");
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE itab3_1_1 () INHERITS (itab3_1)");
+
+$node_subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION mixed CONNECTION '$publisher_connstr' PUBLICATION ipub3, ipub3_1");
+$node_subscriber2->wait_for_subscription_sync;
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3");
+is($result, qq(1|itab3), 'initial data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1), 'initial data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
+is($result, qq(), 'initial data routed for itab3_1_1 on subscriber 2');
+
+# make sure new data is also correctly routed to the roots
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3 VALUES (4, 'itab3_new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1 VALUES (4, 'itab3_1_new')");
+$node_publisher->safe_psql('postgres', "INSERT INTO itab3_1_1 VALUES (4, 'itab3_1_1_new')");
+
+$node_publisher->wait_for_catchup('mixed');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3 ORDER BY 1, 2");
+is($result, qq(1|itab3
+4|itab3_new), 'new data routed for itab3 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1 ORDER BY 1, 2");
+is($result, qq(2|itab3_1
+3|itab3_1_1
+4|itab3_1_1_new
+4|itab3_1_new), 'new data routed for itab3_1 on subscriber 2');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a, b FROM ONLY itab3_1_1");
+is($result, qq(), 'new data routed for itab3_1_1 on subscriber 2');
+
done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 18+ messages in thread
* [PATCH v46 2/7] Add CONCURRENTLY option to REPACK command.
@ 2026-03-11 14:16 Antonin Houska <[email protected]>
0 siblings, 0 replies; 18+ messages in thread
From: Antonin Houska @ 2026-03-11 14:16 UTC (permalink / raw)
The REPACK command copies the relation data into a new file, creates new
indexes and eventually swaps the files. To make sure that the old file does
not change during the copying, the relation is locked in an exclusive mode,
which prevents applications from both reading and writing. (To keep the data
consistent, we'd only need to prevent the applications from writing, but even
reading needs to be blocked before we can swap the files - otherwise some
applications could continue using the old file. Currently, REPACK takes the
simple approach and acquires the exclusive lock in the beginning.
This patch introduces an alternative workflow, which only requires the
exclusive lock when the relation (and index) files are being swapped.
(Supposedly, the swapping should be pretty fast.) On the other hand, when we
copy the data to the new file, we allow applications to read from the relation
and even to write to it.
First, we scan the relation using a "historic snapshot", and insert all the
tuples satisfying this snapshot into the new relation.
Second, logical decoding is used to capture the data changes done by
applications during the copying (i.e. changes not yet committed from the
perspective of the historic snapshot mentioned above), and those are applied
to the new file before we acquire the exclusive lock that we need to swap the
files. (Of course, more data changes can take place while we are waiting for
the lock - these will be applied to the new file after we have acquired the
lock and before we swap the files.)
While the "concurrent data" changes are applied at specific stages (we cannot
do that until the intial copy is finished and indexes are built), a background
worker performs the decoding all the time. This way we minimize the amount of
not-yet-decoded WAL, so that archiving / recycling of WAL segments is not
delayed much. The decoded changes are written to files and passed to the
backed performing REPACK.
Since the logical decoding system, during its startup, waits until all the
transactions which already have XID assigned have finished, there is a risk of
deadlock if a transaction that already changed anything in the database tries
to acquire a conflicting lock on the table REPACK CONCURRENTLY is working
on. As an example, consider transaction running CREATE INDEX command on the
table that is being REPACKed CONCURRENTLY. On the other hand, DML commands
(INSERT, UPDATE, DELETE) are not a problem as their lock does not conflict
with REPACK CONCURRENTLY.
The current approach is that we accept the risk. If we tried to avoid it, it'd
be necessary to unlock the table before the logical decoding is setup and lock
it again afterwards. Such temporary unlocking would imply re-checking if the
table still meets all the requirements for REPACK CONCURRENTLY.
The WAL records produced by running DML commands on the new relation are
intentionally not fed to the logical decoding system. Doing so would introduce
significant overhead, and - as the new relation is never available for logical
replication - it would be useless.
---
doc/src/sgml/monitoring.sgml | 37 +-
doc/src/sgml/mvcc.sgml | 12 +-
doc/src/sgml/ref/repack.sgml | 112 +-
src/Makefile | 1 +
src/backend/access/heap/heapam.c | 41 +-
src/backend/access/heap/heapam_handler.c | 221 +-
src/backend/access/heap/rewriteheap.c | 6 +-
src/backend/access/table/tableam.c | 3 +-
src/backend/catalog/system_views.sql | 19 +-
src/backend/commands/Makefile | 1 +
src/backend/commands/cluster.c | 1823 ++++++++++++++++-
src/backend/commands/matview.c | 1 +
src/backend/commands/meson.build | 1 +
src/backend/commands/repack_worker.c | 546 +++++
src/backend/commands/tablecmds.c | 1 +
src/backend/commands/vacuum.c | 12 +-
src/backend/executor/nodeModifyTable.c | 11 +-
src/backend/libpq/pqmq.c | 5 +
src/backend/meson.build | 1 +
src/backend/postmaster/bgworker.c | 6 +-
src/backend/replication/logical/decode.c | 37 +-
src/backend/replication/logical/logical.c | 6 +-
src/backend/replication/logical/snapbuild.c | 11 +-
.../replication/pgoutput_repack/Makefile | 32 +
.../replication/pgoutput_repack/meson.build | 18 +
.../pgoutput_repack/pgoutput_repack.c | 290 +++
src/backend/replication/walsender.c | 2 +-
src/backend/storage/ipc/procsignal.c | 4 +
.../storage/lmgr/generate-lwlocknames.pl | 2 +-
src/backend/tcop/postgres.c | 4 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/time/snapmgr.c | 3 +-
src/bin/psql/tab-complete.in.c | 4 +-
src/include/access/heapam.h | 6 +-
src/include/access/heapam_xlog.h | 2 +
src/include/access/tableam.h | 39 +-
src/include/commands/cluster.h | 16 +-
src/include/commands/progress.h | 17 +-
src/include/commands/repack_internal.h | 128 ++
src/include/replication/decode.h | 4 +
src/include/replication/snapbuild.h | 2 +-
src/include/storage/lockdefs.h | 4 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/snapmgr.h | 2 +
src/test/modules/injection_points/Makefile | 2 +
.../injection_points/expected/repack.out | 113 +
.../expected/repack_toast.out | 65 +
src/test/modules/injection_points/meson.build | 2 +
.../injection_points/specs/repack.spec | 142 ++
.../injection_points/specs/repack_toast.spec | 112 +
src/test/regress/expected/rules.out | 19 +-
src/tools/pgindent/typedefs.list | 6 +
52 files changed, 3664 insertions(+), 292 deletions(-)
create mode 100644 src/backend/commands/repack_worker.c
create mode 100644 src/backend/replication/pgoutput_repack/Makefile
create mode 100644 src/backend/replication/pgoutput_repack/meson.build
create mode 100644 src/backend/replication/pgoutput_repack/pgoutput_repack.c
create mode 100644 src/include/commands/repack_internal.h
create mode 100644 src/test/modules/injection_points/expected/repack.out
create mode 100644 src/test/modules/injection_points/expected/repack_toast.out
create mode 100644 src/test/modules/injection_points/specs/repack.spec
create mode 100644 src/test/modules/injection_points/specs/repack_toast.spec
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bb75ed1069b..a0a155eb10b 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6985,14 +6985,35 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>heap_tuples_written</structfield> <type>bigint</type>
+ <structfield>heap_tuples_inserted</structfield> <type>bigint</type>
</para>
<para>
- Number of heap tuples written.
+ Number of heap tuples inserted.
This counter only advances when the phase is
<literal>seq scanning heap</literal>,
- <literal>index scanning heap</literal>
- or <literal>writing new heap</literal>.
+ <literal>index scanning heap</literal>,
+ <literal>writing new heap</literal>
+ or <literal>catch-up</literal>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>heap_tuples_updated</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of heap tuples updated.
+ This counter only advances when the phase is <literal>catch-up</literal>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>heap_tuples_deleted</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of heap tuples deleted.
+ This counter only advances when the phase is <literal>catch-up</literal>.
</para></entry>
</row>
@@ -7073,6 +7094,14 @@ FROM pg_stat_get_backend_idset() AS backendid;
<command>REPACK</command> is currently writing the new heap.
</entry>
</row>
+ <row>
+ <entry><literal>catch-up</literal></entry>
+ <entry>
+ <command>REPACK CONCURRENTLY</command> is currently processing the DML
+ commands that other transactions executed during any of the preceding
+ phases.
+ </entry>
+ </row>
<row>
<entry><literal>swapping relation files</literal></entry>
<entry>
diff --git a/doc/src/sgml/mvcc.sgml b/doc/src/sgml/mvcc.sgml
index e775260936a..241caeb3593 100644
--- a/doc/src/sgml/mvcc.sgml
+++ b/doc/src/sgml/mvcc.sgml
@@ -1845,15 +1845,17 @@ SELECT pg_advisory_lock(q.id) FROM
<title>Caveats</title>
<para>
- Some DDL commands, currently only <link linkend="sql-truncate"><command>TRUNCATE</command></link> and the
- table-rewriting forms of <link linkend="sql-altertable"><command>ALTER TABLE</command></link>, are not
+ Some commands, currently only <link linkend="sql-truncate"><command>TRUNCATE</command></link>, the
+ table-rewriting forms of <link linkend="sql-altertable"><command>ALTER
+ TABLE</command></link> and <command>REPACK</command> with
+ the <literal>CONCURRENTLY</literal> option, are not
MVCC-safe. This means that after the truncation or rewrite commits, the
table will appear empty to concurrent transactions, if they are using a
- snapshot taken before the DDL command committed. This will only be an
+ snapshot taken before the command committed. This will only be an
issue for a transaction that did not access the table in question
- before the DDL command started — any transaction that has done so
+ before the command started — any transaction that has done so
would hold at least an <literal>ACCESS SHARE</literal> table lock,
- which would block the DDL command until that transaction completes.
+ which would block the truncating or rewriting command until that transaction completes.
So these commands will not cause any apparent inconsistency in the
table contents for successive queries on the target table, but they
could cause visible inconsistency between the contents of the target
diff --git a/doc/src/sgml/ref/repack.sgml b/doc/src/sgml/ref/repack.sgml
index 8ccf7c7a417..170eb84bf43 100644
--- a/doc/src/sgml/ref/repack.sgml
+++ b/doc/src/sgml/ref/repack.sgml
@@ -28,6 +28,7 @@ REPACK [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] USING
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
ANALYZE [ <replaceable class="parameter">boolean</replaceable> ]
+ CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
<phrase>and <replaceable class="parameter">table_and_columns</replaceable> is:</phrase>
@@ -54,7 +55,8 @@ REPACK [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] USING
processes every table and materialized view in the current database that
the current user has the <literal>MAINTAIN</literal> privilege on. This
form of <command>REPACK</command> cannot be executed inside a transaction
- block.
+ block. Also, this form is not allowed if
+ the <literal>CONCURRENTLY</literal> option is used.
</para>
<para>
@@ -67,7 +69,8 @@ REPACK [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] USING
When a table is being repacked, an <literal>ACCESS EXCLUSIVE</literal> lock
is acquired on it. This prevents any other database operations (both reads
and writes) from operating on the table until the <command>REPACK</command>
- is finished.
+ is finished. If you want to keep the table accessible during the repacking,
+ consider using the <literal>CONCURRENTLY</literal> option.
</para>
<refsect2 id="sql-repack-notes-on-clustering" xreflabel="Notes on Clustering">
@@ -198,6 +201,111 @@ REPACK [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] USING
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>CONCURRENTLY</literal></term>
+ <listitem>
+ <para>
+ Allow other transactions to use the table while it is being repacked.
+ </para>
+
+ <para>
+ Internally, <command>REPACK</command> copies the contents of the table
+ (ignoring dead tuples) into a new file, sorted by the specified index,
+ and also creates a new file for each index. Then it swaps the old and
+ new files for the table and all the indexes, and deletes the old
+ files. The <literal>ACCESS EXCLUSIVE</literal> lock is needed to make
+ sure that the old files do not change during the processing because the
+ changes would get lost due to the swap.
+ </para>
+
+ <para>
+ With the <literal>CONCURRENTLY</literal> option, the <literal>ACCESS
+ EXCLUSIVE</literal> lock is only acquired to swap the table and index
+ files. The data changes that took place during the creation of the new
+ table and index files are captured using logical decoding
+ (<xref linkend="logicaldecoding"/>) and applied before
+ the <literal>ACCESS EXCLUSIVE</literal> lock is requested. Thus the lock
+ is typically held only for the time needed to swap the files, which
+ should be pretty short. However, the time might still be noticeable if
+ too many data changes have been done to the table while
+ <command>REPACK</command> was waiting for the lock: those changes must
+ be processed just before the files are swapped, while the
+ <literal>ACCESS EXCLUSIVE</literal> lock is being held.
+ </para>
+
+ <para>
+ Note that <command>REPACK</command> with the
+ <literal>CONCURRENTLY</literal> option does not try to order the rows
+ inserted into the table after the repacking started. Also
+ note <command>REPACK</command> might fail to complete due to DDL
+ commands executed on the table by other transactions during the
+ repacking.
+ </para>
+
+ <note>
+ <para>
+ In addition to the temporary space requirements explained in
+ <xref linkend="sql-repack-notes-on-resources"/>,
+ the <literal>CONCURRENTLY</literal> option can add to the usage of
+ temporary space a bit more. The reason is that other transactions can
+ perform DML operations which cannot be applied to the new file until
+ <command>REPACK</command> has copied all the existing tuples from the
+ old file. Thus the tuples inserted into the old file during the copying
+ are also stored separately in a temporary file, until they can be
+ processed.
+ </para>
+ </note>
+
+ <para>
+ The <literal>CONCURRENTLY</literal> option cannot be used in the
+ following cases:
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ The table is <literal>UNLOGGED</literal>.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The table is partitioned.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The table is a system catalog or a <acronym>TOAST</acronym> table.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <command>REPACK</command> is executed inside a transaction block.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The <link linkend="guc-max-replication-slots"><varname>max_replication_slots</varname></link>
+ configuration parameter does not allow for creation of an additional
+ replication slot.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <warning>
+ <para>
+ <command>REPACK</command> with the <literal>CONCURRENTLY</literal>
+ option is not MVCC-safe, see <xref linkend="mvcc-caveats"/> for
+ details.
+ </para>
+ </warning>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>VERBOSE</literal></term>
<listitem>
diff --git a/src/Makefile b/src/Makefile
index 2f31a2f20a7..b18c9a14ffa 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -23,6 +23,7 @@ SUBDIRS = \
interfaces \
backend/replication/libpqwalreceiver \
backend/replication/pgoutput \
+ backend/replication/pgoutput_repack \
fe_utils \
bin \
pl \
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index eb1f67f31cd..2ebf2770077 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -61,7 +61,8 @@ static HeapTuple heap_prepare_insert(Relation relation, HeapTuple tup,
static XLogRecPtr log_heap_update(Relation reln, Buffer oldbuf,
Buffer newbuf, HeapTuple oldtup,
HeapTuple newtup, HeapTuple old_key_tuple,
- bool all_visible_cleared, bool new_all_visible_cleared);
+ bool all_visible_cleared, bool new_all_visible_cleared,
+ bool walLogical);
#ifdef USE_ASSERT_CHECKING
static void check_lock_if_inplace_updateable_rel(Relation relation,
const ItemPointerData *otid,
@@ -2850,8 +2851,8 @@ xmax_infomask_changed(uint16 new_infomask, uint16 old_infomask)
*/
TM_Result
heap_delete(Relation relation, const ItemPointerData *tid,
- CommandId cid, Snapshot crosscheck, bool wait,
- TM_FailureData *tmfd, bool changingPart)
+ CommandId cid, Snapshot crosscheck, int options,
+ TM_FailureData *tmfd)
{
TM_Result result;
TransactionId xid = GetCurrentTransactionId();
@@ -2861,6 +2862,9 @@ heap_delete(Relation relation, const ItemPointerData *tid,
BlockNumber block;
Buffer buffer;
Buffer vmbuffer = InvalidBuffer;
+ bool wait = (options & TABLE_DELETE_WAIT) != 0;
+ bool changingPart = (options & TABLE_DELETE_CHANGING_PART) != 0;
+ bool walLogical = (options & TABLE_DELETE_NO_LOGICAL) == 0;
TransactionId new_xmax;
uint16 new_infomask,
new_infomask2;
@@ -3098,7 +3102,8 @@ l1:
* Compute replica identity tuple before entering the critical section so
* we don't PANIC upon a memory allocation failure.
*/
- old_key_tuple = ExtractReplicaIdentity(relation, &tp, true, &old_key_copied);
+ old_key_tuple = walLogical ?
+ ExtractReplicaIdentity(relation, &tp, true, &old_key_copied) : NULL;
/*
* If this is the first possibly-multixact-able operation in the current
@@ -3188,6 +3193,15 @@ l1:
xlrec.flags |= XLH_DELETE_CONTAINS_OLD_KEY;
}
+ /*
+ * Unlike UPDATE, DELETE is decoded even if there is no old key, so it
+ * does not help to clear both XLH_DELETE_CONTAINS_OLD_TUPLE and
+ * XLH_DELETE_CONTAINS_OLD_KEY. Thus we need an extra flag. TODO
+ * Consider not decoding tuples w/o the old tuple/key instead.
+ */
+ if (!walLogical)
+ xlrec.flags |= XLH_DELETE_NO_LOGICAL;
+
XLogBeginInsert();
XLogRegisterData(&xlrec, SizeOfHeapDelete);
@@ -3279,8 +3293,8 @@ simple_heap_delete(Relation relation, const ItemPointerData *tid)
result = heap_delete(relation, tid,
GetCurrentCommandId(true), InvalidSnapshot,
- true /* wait for commit */ ,
- &tmfd, false /* changingPart */ );
+ TABLE_UPDATE_WAIT,
+ &tmfd);
switch (result)
{
case TM_SelfModified:
@@ -3319,7 +3333,7 @@ simple_heap_delete(Relation relation, const ItemPointerData *tid)
*/
TM_Result
heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
- CommandId cid, Snapshot crosscheck, bool wait,
+ CommandId cid, Snapshot crosscheck, int options,
TM_FailureData *tmfd, LockTupleMode *lockmode,
TU_UpdateIndexes *update_indexes)
{
@@ -3336,6 +3350,8 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
HeapTuple heaptup;
HeapTuple old_key_tuple = NULL;
bool old_key_copied = false;
+ bool wait = (options & TABLE_UPDATE_WAIT) != 0;
+ bool walLogical = (options & TABLE_UPDATE_NO_LOGICAL) == 0;
Page page,
newpage;
BlockNumber block;
@@ -4217,7 +4233,8 @@ l2:
newbuf, &oldtup, heaptup,
old_key_tuple,
all_visible_cleared,
- all_visible_cleared_new);
+ all_visible_cleared_new,
+ walLogical);
if (newbuf != buffer)
{
PageSetLSN(newpage, recptr);
@@ -4574,7 +4591,7 @@ simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tup
result = heap_update(relation, otid, tup,
GetCurrentCommandId(true), InvalidSnapshot,
- true /* wait for commit */ ,
+ TABLE_UPDATE_WAIT,
&tmfd, &lockmode, update_indexes);
switch (result)
{
@@ -8892,7 +8909,8 @@ static XLogRecPtr
log_heap_update(Relation reln, Buffer oldbuf,
Buffer newbuf, HeapTuple oldtup, HeapTuple newtup,
HeapTuple old_key_tuple,
- bool all_visible_cleared, bool new_all_visible_cleared)
+ bool all_visible_cleared, bool new_all_visible_cleared,
+ bool walLogical)
{
xl_heap_update xlrec;
xl_heap_header xlhdr;
@@ -8903,7 +8921,8 @@ log_heap_update(Relation reln, Buffer oldbuf,
suffixlen = 0;
XLogRecPtr recptr;
Page page = BufferGetPage(newbuf);
- bool need_tuple_data = RelationIsLogicallyLogged(reln);
+ bool need_tuple_data = RelationIsLogicallyLogged(reln) &&
+ walLogical;
bool init;
int bufflags;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index d40878928e1..0ba7c0df1fc 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -311,22 +311,22 @@ heapam_tuple_complete_speculative(Relation relation, TupleTableSlot *slot,
static TM_Result
heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid,
- Snapshot snapshot, Snapshot crosscheck, bool wait,
- TM_FailureData *tmfd, bool changingPart)
+ Snapshot snapshot, Snapshot crosscheck, int options,
+ TM_FailureData *tmfd)
{
/*
* Currently Deleting of index tuples are handled at vacuum, in case if
* the storage itself is cleaning the dead tuples by itself, it is the
* time to call the index tuple deletion also.
*/
- return heap_delete(relation, tid, cid, crosscheck, wait, tmfd, changingPart);
+ return heap_delete(relation, tid, cid, crosscheck, options, tmfd);
}
static TM_Result
heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot,
CommandId cid, Snapshot snapshot, Snapshot crosscheck,
- bool wait, TM_FailureData *tmfd,
+ int options, TM_FailureData *tmfd,
LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes)
{
bool shouldFree = true;
@@ -337,7 +337,7 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot,
slot->tts_tableOid = RelationGetRelid(relation);
tuple->t_tableOid = slot->tts_tableOid;
- result = heap_update(relation, otid, tuple, cid, crosscheck, wait,
+ result = heap_update(relation, otid, tuple, cid, crosscheck, options,
tmfd, lockmode, update_indexes);
ItemPointerCopy(&tuple->t_self, &slot->tts_tid);
@@ -695,13 +695,14 @@ static void
heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
Relation OldIndex, bool use_sort,
TransactionId OldestXmin,
+ Snapshot snapshot,
TransactionId *xid_cutoff,
MultiXactId *multi_cutoff,
double *num_tuples,
double *tups_vacuumed,
double *tups_recently_dead)
{
- RewriteState rwstate;
+ RewriteState rwstate = NULL;
IndexScanDesc indexScan;
TableScanDesc tableScan;
HeapScanDesc heapScan;
@@ -715,6 +716,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
bool *isnull;
BufferHeapTupleTableSlot *hslot;
BlockNumber prev_cblock = InvalidBlockNumber;
+ bool concurrent = snapshot != NULL;
/* Remember if it's a system catalog */
is_system_catalog = IsSystemRelation(OldHeap);
@@ -730,9 +732,12 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
values = palloc_array(Datum, natts);
isnull = palloc_array(bool, natts);
- /* Initialize the rewrite operation */
- rwstate = begin_heap_rewrite(OldHeap, NewHeap, OldestXmin, *xid_cutoff,
- *multi_cutoff);
+ /*
+ * Initialize the rewrite operation.
+ */
+ if (!concurrent)
+ rwstate = begin_heap_rewrite(OldHeap, NewHeap, OldestXmin,
+ *xid_cutoff, *multi_cutoff);
/* Set up sorting if wanted */
@@ -747,6 +752,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
* Prepare to scan the OldHeap. To ensure we see recently-dead tuples
* that still need to be copied, we scan with SnapshotAny and use
* HeapTupleSatisfiesVacuum for the visibility test.
+ *
+ * In the CONCURRENTLY case, we do regular MVCC visibility tests, using
+ * the snapshot passed by the caller.
*/
if (OldIndex != NULL && !use_sort)
{
@@ -763,7 +771,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
tableScan = NULL;
heapScan = NULL;
- indexScan = index_beginscan(OldHeap, OldIndex, SnapshotAny, NULL, 0, 0);
+ indexScan = index_beginscan(OldHeap, OldIndex,
+ snapshot ? snapshot : SnapshotAny,
+ NULL, 0, 0);
index_rescan(indexScan, NULL, 0, NULL, 0);
}
else
@@ -772,7 +782,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
PROGRESS_REPACK_PHASE_SEQ_SCAN_HEAP);
- tableScan = table_beginscan(OldHeap, SnapshotAny, 0, (ScanKey) NULL);
+ tableScan = table_beginscan(OldHeap,
+ snapshot ? snapshot : SnapshotAny,
+ 0, (ScanKey) NULL);
heapScan = (HeapScanDesc) tableScan;
indexScan = NULL;
@@ -848,83 +860,91 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
buf = hslot->buffer;
/*
- * To be able to guarantee that we can set the hint bit, acquire an
- * exclusive lock on the old buffer. We need the hint bits, set in
- * heapam_relation_copy_for_cluster() -> HeapTupleSatisfiesVacuum(),
- * to be set, as otherwise reform_and_rewrite_tuple() ->
- * rewrite_heap_tuple() will get confused. Specifically,
- * rewrite_heap_tuple() checks for HEAP_XMAX_INVALID in the old tuple
- * to determine whether to check the old-to-new mapping hash table.
- *
- * It'd be better if we somehow could avoid setting hint bits on the
- * old page. One reason to use VACUUM FULL are very bloated tables -
- * rewriting most of the old table during VACUUM FULL doesn't exactly
- * help...
+ * Regarding CONCURRENTLY, see the comments on MVCC snapshot above.
*/
- LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
-
- switch (HeapTupleSatisfiesVacuum(tuple, OldestXmin, buf))
+ if (!concurrent)
{
- case HEAPTUPLE_DEAD:
- /* Definitely dead */
- isdead = true;
- break;
- case HEAPTUPLE_RECENTLY_DEAD:
- *tups_recently_dead += 1;
- pg_fallthrough;
- case HEAPTUPLE_LIVE:
- /* Live or recently dead, must copy it */
- isdead = false;
- break;
- case HEAPTUPLE_INSERT_IN_PROGRESS:
+ /*
+ * To be able to guarantee that we can set the hint bit, acquire
+ * an exclusive lock on the old buffer. We need the hint bits, set
+ * in heapam_relation_copy_for_cluster() ->
+ * HeapTupleSatisfiesVacuum(), to be set, as otherwise
+ * reform_and_rewrite_tuple() -> rewrite_heap_tuple() will get
+ * confused. Specifically, rewrite_heap_tuple() checks for
+ * HEAP_XMAX_INVALID in the old tuple to determine whether to
+ * check the old-to-new mapping hash table.
+ *
+ * It'd be better if we somehow could avoid setting hint bits on
+ * the old page. One reason to use VACUUM FULL are very bloated
+ * tables - rewriting most of the old table during VACUUM FULL
+ * doesn't exactly help...
+ */
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- /*
- * Since we hold exclusive lock on the relation, normally the
- * only way to see this is if it was inserted earlier in our
- * own transaction. However, it can happen in system
- * catalogs, since we tend to release write lock before commit
- * there. Give a warning if neither case applies; but in any
- * case we had better copy it.
- */
- if (!is_system_catalog &&
- !TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tuple->t_data)))
- elog(WARNING, "concurrent insert in progress within table \"%s\"",
- RelationGetRelationName(OldHeap));
- /* treat as live */
- isdead = false;
- break;
- case HEAPTUPLE_DELETE_IN_PROGRESS:
-
- /*
- * Similar situation to INSERT_IN_PROGRESS case.
- */
- if (!is_system_catalog &&
- !TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetUpdateXid(tuple->t_data)))
- elog(WARNING, "concurrent delete in progress within table \"%s\"",
- RelationGetRelationName(OldHeap));
- /* treat as recently dead */
- *tups_recently_dead += 1;
- isdead = false;
- break;
- default:
- elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
- isdead = false; /* keep compiler quiet */
- break;
- }
-
- LockBuffer(buf, BUFFER_LOCK_UNLOCK);
-
- if (isdead)
- {
- *tups_vacuumed += 1;
- /* heap rewrite module still needs to see it... */
- if (rewrite_heap_dead_tuple(rwstate, tuple))
+ switch (HeapTupleSatisfiesVacuum(tuple, OldestXmin, buf))
{
- /* A previous recently-dead tuple is now known dead */
- *tups_vacuumed += 1;
- *tups_recently_dead -= 1;
+ case HEAPTUPLE_DEAD:
+ /* Definitely dead */
+ isdead = true;
+ break;
+ case HEAPTUPLE_RECENTLY_DEAD:
+ *tups_recently_dead += 1;
+ pg_fallthrough;
+ case HEAPTUPLE_LIVE:
+ /* Live or recently dead, must copy it */
+ isdead = false;
+ break;
+ case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+ /*
+ * As long as we hold exclusive lock on the relation,
+ * normally the only way to see this is if it was inserted
+ * earlier in our own transaction. However, it can happen
+ * in system catalogs, since we tend to release write lock
+ * before commit there. Give a warning if neither case
+ * applies; but in any case we had better copy it.
+ */
+ if (!is_system_catalog &&
+ !TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(tuple->t_data)))
+ elog(WARNING, "concurrent insert in progress within table \"%s\"",
+ RelationGetRelationName(OldHeap));
+ /* treat as live */
+ isdead = false;
+ break;
+ case HEAPTUPLE_DELETE_IN_PROGRESS:
+
+ /*
+ * Similar situation to INSERT_IN_PROGRESS case.
+ */
+ if (!is_system_catalog &&
+ !TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetUpdateXid(tuple->t_data)))
+ elog(WARNING, "concurrent delete in progress within table \"%s\"",
+ RelationGetRelationName(OldHeap));
+ /* treat as recently dead */
+ *tups_recently_dead += 1;
+ isdead = false;
+ break;
+ default:
+ elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+ isdead = false; /* keep compiler quiet */
+ break;
+ }
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ if (isdead)
+ {
+ *tups_vacuumed += 1;
+ /* heap rewrite module still needs to see it... */
+ if (rewrite_heap_dead_tuple(rwstate, tuple))
+ {
+ /* A previous recently-dead tuple is now known dead */
+ *tups_vacuumed += 1;
+ *tups_recently_dead -= 1;
+ }
+
+ continue;
}
- continue;
}
*num_tuples += 1;
@@ -943,7 +963,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
{
const int ct_index[] = {
PROGRESS_REPACK_HEAP_TUPLES_SCANNED,
- PROGRESS_REPACK_HEAP_TUPLES_WRITTEN
+ PROGRESS_REPACK_HEAP_TUPLES_INSERTED
};
int64 ct_val[2];
@@ -1001,7 +1021,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
values, isnull,
rwstate);
/* Report n_tuples */
- pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_WRITTEN,
+ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED,
n_tuples);
}
@@ -1009,7 +1029,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
}
/* Write out any remaining tuples, and fsync if needed */
- end_heap_rewrite(rwstate);
+ if (rwstate)
+ end_heap_rewrite(rwstate);
/* Clean up */
pfree(values);
@@ -2402,6 +2423,10 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
* SET WITHOUT OIDS.
*
* So, we must reconstruct the tuple from component Datums.
+ *
+ * If rwstate=NULL, use simple_heap_insert() instead of rewriting - in that
+ * case we still need to deform/form the tuple. TODO Shouldn't we rename the
+ * function, as might not do any rewrite?
*/
static void
reform_and_rewrite_tuple(HeapTuple tuple,
@@ -2424,8 +2449,28 @@ reform_and_rewrite_tuple(HeapTuple tuple,
copiedTuple = heap_form_tuple(newTupDesc, values, isnull);
- /* The heap rewrite module does the rest */
- rewrite_heap_tuple(rwstate, tuple, copiedTuple);
+ if (rwstate)
+ /* The heap rewrite module does the rest */
+ rewrite_heap_tuple(rwstate, tuple, copiedTuple);
+ else
+ {
+ /*
+ * Insert tuple when processing REPACK CONCURRENTLY.
+ *
+ * rewriteheap.c is not used in the CONCURRENTLY case because it'd be
+ * difficult to do the same in the catch-up phase (as the logical
+ * decoding does not provide us with sufficient visibility
+ * information). Thus we must use heap_insert() both during the
+ * catch-up and here.
+ *
+ * The following is like simple_heap_insert() except that we pass the
+ * flag to skip logical decoding: as soon as REPACK CONCURRENTLY swaps
+ * the relation files, it drops this relation, so no logical
+ * replication subscription should need the data.
+ */
+ heap_insert(NewHeap, copiedTuple, GetCurrentCommandId(true),
+ HEAP_INSERT_NO_LOGICAL, NULL);
+ }
heap_freetuple(copiedTuple);
}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 6b19ac3030d..d706856e7a5 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -621,9 +621,9 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
int options = HEAP_INSERT_SKIP_FSM;
/*
- * While rewriting the heap for VACUUM FULL / CLUSTER, make sure data
- * for the TOAST table are not logically decoded. The main heap is
- * WAL-logged as XLOG FPI records, which are not logically decoded.
+ * While rewriting the heap for REPACK, make sure data for the TOAST
+ * table are not logically decoded. The main heap is WAL-logged as
+ * XLOG FPI records, which are not logically decoded.
*/
options |= HEAP_INSERT_NO_LOGICAL;
diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.c
index dfda1af412e..4c759f29b3e 100644
--- a/src/backend/access/table/tableam.c
+++ b/src/backend/access/table/tableam.c
@@ -319,8 +319,7 @@ simple_table_tuple_delete(Relation rel, ItemPointer tid, Snapshot snapshot)
result = table_tuple_delete(rel, tid,
GetCurrentCommandId(true),
snapshot, InvalidSnapshot,
- true /* wait for commit */ ,
- &tmfd, false /* changingPart */ );
+ TABLE_DELETE_WAIT, &tmfd);
switch (result)
{
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index e54018004db..75f97cb23e3 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1343,16 +1343,19 @@ CREATE VIEW pg_stat_progress_repack AS
WHEN 2 THEN 'index scanning heap'
WHEN 3 THEN 'sorting tuples'
WHEN 4 THEN 'writing new heap'
- WHEN 5 THEN 'swapping relation files'
- WHEN 6 THEN 'rebuilding index'
- WHEN 7 THEN 'performing final cleanup'
+ WHEN 5 THEN 'catch-up'
+ WHEN 6 THEN 'swapping relation files'
+ WHEN 7 THEN 'rebuilding index'
+ WHEN 8 THEN 'performing final cleanup'
END AS phase,
CAST(S.param3 AS oid) AS repack_index_relid,
S.param4 AS heap_tuples_scanned,
- S.param5 AS heap_tuples_written,
- S.param6 AS heap_blks_total,
- S.param7 AS heap_blks_scanned,
- S.param8 AS index_rebuild_count
+ S.param5 AS heap_tuples_inserted,
+ S.param6 AS heap_tuples_updated,
+ S.param7 AS heap_tuples_deleted,
+ S.param8 AS heap_blks_total,
+ S.param9 AS heap_blks_scanned,
+ S.param10 AS index_rebuild_count
FROM pg_stat_get_progress_info('REPACK') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
@@ -1370,7 +1373,7 @@ CREATE VIEW pg_stat_progress_cluster AS
phase,
repack_index_relid AS cluster_index_relid,
heap_tuples_scanned,
- heap_tuples_written,
+ heap_tuples_inserted + heap_tuples_updated AS heap_tuples_written,
heap_blks_total,
heap_blks_scanned,
index_rebuild_count
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index c10fdba2bbb..6926bc44818 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -51,6 +51,7 @@ OBJS = \
proclang.o \
propgraphcmds.o \
publicationcmds.o \
+ repack_worker.o \
schemacmds.o \
seclabel.o \
sequence.o \
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 09066db0956..33083a2b030 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -4,6 +4,22 @@
* REPACK a table; formerly known as CLUSTER. VACUUM FULL also uses
* parts of this code.
*
+ * There are two somewhat different ways to rewrite a table. In non-
+ * concurrent mode, it's easy: take AccessExclusiveLock, create a new
+ * transient relation, copy the tuples over to the relfilenode of the new
+ * relation, swap the relfilenodes, then drop the old relation.
+ *
+ * In concurrent mode, we lock the table with only ShareUpdateExclusiveLock,
+ * then do an initial copy as above. However, while the tuples are being
+ * copied, concurrent transactions could modify the table. To cope with those
+ * changes, we rely on logical decoding to obtain them from WAL. A bgworker
+ * consumes WAL while the initial copy is ongoing (to prevent excessive WAL
+ * from being reserved), and accumulates the changes in a file. Once the
+ * initial copy is complete, we read the changes from the file and re-apply
+ * them on the new heap. Then we upgrade our ShareUpdateExclusiveLock to
+ * AccessExclusiveLock and swap the relfilenodes. This way, the time we hold
+ * a strong lock on the table is much reduced, and the bloat is eliminated.
+ *
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994-5, Regents of the University of California
@@ -17,6 +33,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/detoast.h"
#include "access/heapam.h"
#include "access/multixact.h"
#include "access/relscan.h"
@@ -24,6 +41,7 @@
#include "access/toast_internals.h"
#include "access/transam.h"
#include "access/xact.h"
+#include "access/xloginsert.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
@@ -31,22 +49,29 @@
#include "catalog/namespace.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_am.h"
+#include "catalog/pg_control.h"
#include "catalog/pg_inherits.h"
#include "catalog/toasting.h"
#include "commands/cluster.h"
#include "commands/defrem.h"
#include "commands/progress.h"
+#include "commands/repack_internal.h"
#include "commands/tablecmds.h"
#include "commands/vacuum.h"
+#include "executor/executor.h"
+#include "libpq/pqformat.h"
+#include "libpq/pqmq.h"
#include "miscadmin.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/lmgr.h"
#include "storage/predicate.h"
+#include "storage/proc.h"
#include "utils/acl.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
+#include "utils/injection_point.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -54,6 +79,7 @@
#include "utils/relmapper.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/wait_event_types.h"
/*
* This struct is used to pass around the information on tables to be
@@ -66,12 +92,79 @@ typedef struct
Oid indexOid;
} RelToCluster;
+/*
+ * The first file exported by the decoding worker must contain a snapshot, the
+ * following ones contain the data changes.
+ */
+#define WORKER_FILE_SNAPSHOT 0
+
+/*
+ * Information needed to apply concurrent data changes.
+ */
+typedef struct ChangeContext
+{
+ /* The relation the changes are applied to. */
+ Relation cc_rel;
+
+ /* Needed to update indexes of rel_dst. */
+ ResultRelInfo *cc_rri;
+ EState *cc_estate;
+
+ /*
+ * Existing tuples to UPDATE and DELETE are located via this index. We
+ * keep the scankey in partially initialized state to avoid repeated work.
+ * sk_argument is completed on the fly.
+ */
+ Relation cc_ident_index;
+ ScanKey cc_ident_key;
+ int cc_ident_key_nentries;
+
+ /* Sequential number of the file containing the changes. */
+ int cc_file_seq;
+} ChangeContext;
+
+/* FIXME probably in repack_worker.h */
+pid_t backend_pid;
+ProcNumber backend_proc_number;
+
+/*
+ * Backend-local information to control the decoding worker.
+ */
+typedef struct DecodingWorker
+{
+ /* The worker. */
+ BackgroundWorkerHandle *handle;
+
+ /* DecodingWorkerShared is in this segment. */
+ dsm_segment *seg;
+
+ /* Handle of the error queue. */
+ shm_mq_handle *error_mqh;
+} DecodingWorker;
+
+/* Pointer to currently running decoding worker. */
+static DecodingWorker *decoding_worker = NULL;
+
+/*
+ * Is there a message sent by a repack worker that the backend needs to
+ * receive?
+ */
+volatile sig_atomic_t RepackMessagePending = false;
+
+static LOCKMODE RepackLockLevel(bool concurrent);
static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap,
- Oid indexOid, Oid userid, int options);
-static void rebuild_relation(Relation OldHeap, Relation index, bool verbose);
+ Oid indexOid, Oid userid, LOCKMODE lmode,
+ int options);
+static void check_repack_concurrently_requirements(Relation rel,
+ Oid *ident_idx_p);
+static void rebuild_relation(Relation OldHeap, Relation index, bool verbose,
+ Oid ident_idx);
static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
- bool verbose, bool *pSwapToastByContent,
- TransactionId *pFreezeXid, MultiXactId *pCutoffMulti);
+ Snapshot snapshot,
+ bool verbose,
+ bool *pSwapToastByContent,
+ TransactionId *pFreezeXid,
+ MultiXactId *pCutoffMulti);
static List *get_tables_to_repack(RepackCommand cmd, bool usingindex,
MemoryContext permcxt);
static List *get_tables_to_repack_partitioned(RepackCommand cmd,
@@ -79,10 +172,45 @@ static List *get_tables_to_repack_partitioned(RepackCommand cmd,
MemoryContext permcxt);
static bool repack_is_permitted_for_relation(RepackCommand cmd,
Oid relid, Oid userid);
+
+static void apply_concurrent_changes(BufFile *file, ChangeContext *chgcxt);
+static void apply_concurrent_insert(Relation rel, TupleTableSlot *slot,
+ ChangeContext *chgcxt);
+static void apply_concurrent_update(Relation rel, TupleTableSlot *spilled_tuple,
+ TupleTableSlot *ondisk_tuple,
+ ChangeContext *chgcxt);
+static void apply_concurrent_delete(Relation rel, TupleTableSlot *slot);
+static void restore_tuple(BufFile *file, Relation relation,
+ TupleTableSlot *slot);
+static void adjust_toast_pointers(Relation relation, TupleTableSlot *dest,
+ TupleTableSlot *src);
+static bool find_target_tuple(Relation rel, ChangeContext *chgcxt,
+ TupleTableSlot *locator,
+ TupleTableSlot *received);
+static void process_concurrent_changes(XLogRecPtr end_of_wal,
+ ChangeContext *chgcxt,
+ bool done);
+static void initialize_change_context(ChangeContext *chgcxt,
+ Relation relation,
+ Oid ident_index_id);
+static void release_change_context(ChangeContext *chgcxt);
+static void rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
+ Oid identIdx,
+ TransactionId frozenXid,
+ MultiXactId cutoffMulti);
+static List *build_new_indexes(Relation NewHeap, Relation OldHeap, List *OldIndexes);
static Relation process_single_relation(RepackStmt *stmt,
+ LOCKMODE lockmode,
+ bool isTopLevel,
ClusterParams *params);
static Oid determine_clustered_index(Relation rel, bool usingindex,
const char *indexname);
+
+static void start_repack_decoding_worker(Oid relid);
+static void stop_repack_decoding_worker(void);
+static Snapshot get_initial_snapshot(DecodingWorker *worker);
+
+static void ProcessRepackMessage(StringInfo msg);
static const char *RepackCommandAsString(RepackCommand cmd);
@@ -115,6 +243,7 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
ClusterParams params = {0};
Relation rel = NULL;
MemoryContext repack_context;
+ LOCKMODE lockmode;
List *rtcs;
/* Parse option list */
@@ -125,6 +254,16 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
else if (strcmp(opt->defname, "analyze") == 0 ||
strcmp(opt->defname, "analyse") == 0)
params.options |= defGetBoolean(opt) ? CLUOPT_ANALYZE : 0;
+ else if (strcmp(opt->defname, "concurrently") == 0 &&
+ defGetBoolean(opt))
+ {
+ if (stmt->command != REPACK_COMMAND_REPACK)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("CONCURRENTLY option not supported for %s",
+ RepackCommandAsString(stmt->command)));
+ params.options |= CLUOPT_CONCURRENT;
+ }
else
ereport(ERROR,
errcode(ERRCODE_SYNTAX_ERROR),
@@ -134,13 +273,16 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
parser_errposition(pstate, opt->location));
}
+ /* Determine the lock mode to use. */
+ lockmode = RepackLockLevel((params.options & CLUOPT_CONCURRENT) != 0);
+
/*
* If a single relation is specified, process it and we're done ... unless
* the relation is a partitioned table, in which case we fall through.
*/
if (stmt->relation != NULL)
{
- rel = process_single_relation(stmt, ¶ms);
+ rel = process_single_relation(stmt, lockmode, isTopLevel, ¶ms);
if (rel == NULL)
return; /* all done */
}
@@ -156,10 +298,29 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
"REPACK (ANALYZE)"));
/*
- * By here, we know we are in a multi-table situation. In order to avoid
- * holding locks for too long, we want to process each table in its own
- * transaction. This forces us to disallow running inside a user
- * transaction block.
+ * By here, we know we are in a multi-table situation.
+ *
+ * Concurrent processing is currently considered rather special (e.g. in
+ * terms of resources consumed) so it is not performed in bulk.
+ */
+ if (params.options & CLUOPT_CONCURRENT)
+ {
+ if (rel != NULL)
+ {
+ Assert(rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE);
+ ereport(ERROR,
+ errmsg("REPACK CONCURRENTLY not supported for partitioned tables"),
+ errhint("Consider running the command for individual partitions."));
+ }
+ else
+ ereport(ERROR,
+ errmsg("REPACK CONCURRENTLY requires explicit table name"));
+ }
+
+ /*
+ * In order to avoid holding locks for too long, we want to process each
+ * table in its own transaction. This forces us to disallow running
+ * inside a user transaction block.
*/
PreventInTransactionBlock(isTopLevel, RepackCommandAsString(stmt->command));
@@ -168,6 +329,12 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
"Repack",
ALLOCSET_DEFAULT_SIZES);
+ /*
+ * Since we open a new transaction for each relation, we have to check
+ * that the relation still is what we think it is.
+ *
+ * In single-transaction CLUSTER, we don't need the overhead.
+ */
params.options |= CLUOPT_RECHECK;
/*
@@ -253,7 +420,7 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
* Open the target table, coping with the case where it has been
* dropped.
*/
- rel = try_table_open(rtc->tableOid, AccessExclusiveLock);
+ rel = try_table_open(rtc->tableOid, lockmode);
if (rel == NULL)
{
CommitTransactionCommand();
@@ -264,7 +431,7 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
PushActiveSnapshot(GetTransactionSnapshot());
/* Process this table */
- cluster_rel(stmt->command, rel, rtc->indexOid, ¶ms);
+ cluster_rel(stmt->command, rel, rtc->indexOid, ¶ms, isTopLevel);
/* cluster_rel closes the relation, but keeps lock */
PopActiveSnapshot();
@@ -278,6 +445,22 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
MemoryContextDelete(repack_context);
}
+/*
+ * In the non-concurrent case, we obtain AccessExclusiveLock throughout the
+ * operation to avoid any lock-upgrade hazards. In the concurrent case, we
+ * grab ShareUpdateExclusiveLock (jsut like VACUUM) for most of the
+ * processing and only acquire AccessExclusiveLock at the end, to swap the
+ * relation -- supposedly for a short time.
+ */
+static LOCKMODE
+RepackLockLevel(bool concurrent)
+{
+ if (concurrent)
+ return ShareUpdateExclusiveLock;
+ else
+ return AccessExclusiveLock;
+}
+
/*
* cluster_rel
*
@@ -293,22 +476,51 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
* If indexOid is InvalidOid, the table will be rewritten in physical order
* instead of index order.
*
+ * Note that, in the concurrent case, the function releases the lock at some
+ * point, in order to get AccessExclusiveLock for the final steps (i.e. to
+ * swap the relation files). To make things simpler, the caller should expect
+ * OldHeap to be closed on return, regardless CLUOPT_CONCURRENT. (The
+ * AccessExclusiveLock is kept till the end of the transaction.)
+ *
* 'cmd' indicates which command is being executed, to be used for error
* messages.
*/
void
cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
- ClusterParams *params)
+ ClusterParams *params, bool isTopLevel)
{
Oid tableOid = RelationGetRelid(OldHeap);
+ Relation index;
+ LOCKMODE lmode;
Oid save_userid;
int save_sec_context;
int save_nestlevel;
bool verbose = ((params->options & CLUOPT_VERBOSE) != 0);
bool recheck = ((params->options & CLUOPT_RECHECK) != 0);
- Relation index;
+ bool concurrent = ((params->options & CLUOPT_CONCURRENT) != 0);
+ Oid ident_idx = InvalidOid;
- Assert(CheckRelationLockedByMe(OldHeap, AccessExclusiveLock, false));
+ /* Determine the lock mode to use. */
+ lmode = RepackLockLevel(concurrent);
+
+ /*
+ * Check some preconditions in the concurrent case. This also obtains the
+ * replica index OID.
+ */
+ if (concurrent)
+ {
+ /*
+ * Make sure we're not in a transaction block.
+ *
+ * The reason is that repack_setup_logical_decoding() could deadlock
+ * if there's an XID already assigned. It would be possible to run in
+ * a transaction block if we had no XID, but this restriction is
+ * simpler for users to understand and we don't lose anything.
+ */
+ PreventInTransactionBlock(isTopLevel, "REPACK (CONCURRENTLY)");
+
+ check_repack_concurrently_requirements(OldHeap, &ident_idx);
+ }
/* Check for user-requested abort. */
CHECK_FOR_INTERRUPTS();
@@ -328,16 +540,15 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
RestrictSearchPath();
/*
- * Since we may open a new transaction for each relation, we have to check
- * that the relation still is what we think it is.
+ * Recheck that the relation is still what it was when we started.
*
- * If this is a single-transaction CLUSTER, we can skip these tests. We
- * *must* skip the one on indisclustered since it would reject an attempt
- * to cluster a not-previously-clustered index.
+ * Note that it's critical to skip this in single-relation CLUSTER;
+ * otherwise, we would reject an attempt to cluster using a
+ * not-previously-clustered index.
*/
if (recheck &&
!cluster_rel_recheck(cmd, OldHeap, indexOid, save_userid,
- params->options))
+ lmode, params->options))
goto out;
/*
@@ -353,6 +564,12 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
errmsg("cannot execute %s on a shared catalog",
RepackCommandAsString(cmd)));
+ /*
+ * The CONCURRENTLY case should have been rejected earlier because it does
+ * not support system catalogs.
+ */
+ Assert(!(OldHeap->rd_rel->relisshared && concurrent));
+
/*
* Don't process temp tables of other backends ... their local buffer
* manager is not going to cope.
@@ -374,7 +591,7 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
if (OidIsValid(indexOid))
{
/* verify the index is good and lock it */
- check_index_is_clusterable(OldHeap, indexOid, AccessExclusiveLock);
+ check_index_is_clusterable(OldHeap, indexOid, lmode);
/* also open it */
index = index_open(indexOid, NoLock);
}
@@ -409,7 +626,9 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
if (OldHeap->rd_rel->relkind == RELKIND_MATVIEW &&
!RelationIsPopulated(OldHeap))
{
- relation_close(OldHeap, AccessExclusiveLock);
+ if (index)
+ index_close(index, lmode);
+ relation_close(OldHeap, lmode);
goto out;
}
@@ -422,11 +641,34 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
* invalid, because we move tuples around. Promote them to relation
* locks. Predicate locks on indexes will be promoted when they are
* reindexed.
+ *
+ * During concurrent processing, the heap as well as its indexes stay in
+ * operation, so we postpone this step until they are locked using
+ * AccessExclusiveLock near the end of the processing.
*/
- TransferPredicateLocksToHeapRelation(OldHeap);
+ if (!concurrent)
+ TransferPredicateLocksToHeapRelation(OldHeap);
/* rebuild_relation does all the dirty work */
- rebuild_relation(OldHeap, index, verbose);
+ PG_TRY();
+ {
+ rebuild_relation(OldHeap, index, verbose, ident_idx);
+ }
+ PG_FINALLY();
+ {
+ if (concurrent)
+ {
+ /*
+ * Since during normal operation the worker was already asked to
+ * exit, stopping it explicitly is especially important on ERROR.
+ * However it still seems a good practice to make sure that the
+ * worker never survives the REPACK command.
+ */
+ stop_repack_decoding_worker();
+ }
+ }
+ PG_END_TRY();
+
/* rebuild_relation closes OldHeap, and index if valid */
out:
@@ -445,14 +687,14 @@ out:
*/
static bool
cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid,
- Oid userid, int options)
+ Oid userid, LOCKMODE lmode, int options)
{
Oid tableOid = RelationGetRelid(OldHeap);
/* Check that the user still has privileges for the relation */
if (!repack_is_permitted_for_relation(cmd, tableOid, userid))
{
- relation_close(OldHeap, AccessExclusiveLock);
+ relation_close(OldHeap, lmode);
return false;
}
@@ -466,7 +708,7 @@ cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid,
*/
if (RELATION_IS_OTHER_TEMP(OldHeap))
{
- relation_close(OldHeap, AccessExclusiveLock);
+ relation_close(OldHeap, lmode);
return false;
}
@@ -477,7 +719,7 @@ cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid,
*/
if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(indexOid)))
{
- relation_close(OldHeap, AccessExclusiveLock);
+ relation_close(OldHeap, lmode);
return false;
}
@@ -488,7 +730,7 @@ cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid,
if ((options & CLUOPT_RECHECK_ISCLUSTERED) != 0 &&
!get_index_isclustered(indexOid))
{
- relation_close(OldHeap, AccessExclusiveLock);
+ relation_close(OldHeap, lmode);
return false;
}
}
@@ -500,7 +742,7 @@ cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid,
* Verify that the specified heap and index are valid to cluster on
*
* Side effect: obtains lock on the index. The caller may
- * in some cases already have AccessExclusiveLock on the table, but
+ * in some cases already have a lock of the same strength on the table, but
* not in all cases so we can't rely on the table-level lock for
* protection here.
*/
@@ -626,17 +868,94 @@ mark_index_clustered(Relation rel, Oid indexOid, bool is_internal)
}
/*
- * rebuild_relation: rebuild an existing relation in index or physical order
+ * Check if the CONCURRENTLY option is legal for the relation.
*
- * OldHeap: table to rebuild.
- * index: index to cluster by, or NULL to rewrite in physical order.
- *
- * On entry, heap and index (if one is given) must be open, and
- * AccessExclusiveLock held on them.
- * On exit, they are closed, but locks on them are not released.
+ * *Ident_idx_p receives OID of the identity index.
*/
static void
-rebuild_relation(Relation OldHeap, Relation index, bool verbose)
+check_repack_concurrently_requirements(Relation rel, Oid *ident_idx_p)
+{
+ char relpersistence,
+ replident;
+ Oid ident_idx;
+
+ /* Data changes in system relations are not logically decoded. */
+ if (IsCatalogRelation(rel))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot repack relation \"%s\"",
+ RelationGetRelationName(rel)),
+ errhint("REPACK CONCURRENTLY is not supported for catalog relations."));
+
+ /*
+ * reorderbuffer.c does not seem to handle processing of TOAST relation
+ * alone.
+ */
+ if (IsToastRelation(rel))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot repack relation \"%s\"",
+ RelationGetRelationName(rel)),
+ errhint("REPACK CONCURRENTLY is not supported for TOAST relations, unless the main relation is repacked too."));
+
+ relpersistence = rel->rd_rel->relpersistence;
+ if (relpersistence != RELPERSISTENCE_PERMANENT)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot repack relation \"%s\"",
+ RelationGetRelationName(rel)),
+ errhint("REPACK CONCURRENTLY is only allowed for permanent relations."));
+
+ /* With NOTHING, WAL does not contain the old tuple. */
+ replident = rel->rd_rel->relreplident;
+ if (replident == REPLICA_IDENTITY_NOTHING)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot repack relation \"%s\"",
+ RelationGetRelationName(rel)),
+ errhint("Relation \"%s\" has insufficient replication identity.",
+ RelationGetRelationName(rel)));
+
+ /*
+ * If the identity index is not set due to replica identity being, PK
+ * might exist.
+ */
+ ident_idx = RelationGetReplicaIndex(rel);
+ if (!OidIsValid(ident_idx) && OidIsValid(rel->rd_pkindex))
+ ident_idx = rel->rd_pkindex;
+ if (!OidIsValid(ident_idx))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot process relation \"%s\"",
+ RelationGetRelationName(rel)),
+ errhint("Relation \"%s\" has no identity index.",
+ RelationGetRelationName(rel)));
+
+ *ident_idx_p = ident_idx;
+}
+
+
+/*
+ * rebuild_relation: rebuild an existing relation in index or physical order
+ *
+ * OldHeap: table to rebuild. See cluster_rel() for comments on the required
+ * lock strength.
+ *
+ * index: index to cluster by, or NULL to rewrite in physical order.
+ *
+ * ident_idx: identity index, to handle replaying of concurrent data changes
+ * to the new heap. InvalidOid if there's no CONCURRENTLY option.
+ *
+ * On entry, heap and index (if one is given) must be open, and the
+ * appropriate lock held on them -- AccessExclusiveLock for exclusive
+ * processing and ShareUpdateExclusiveLock for concurrent processing.
+ *
+ * On exit, they are closed, but still locked with AccessExclusiveLock.
+ * (The function handles the lock upgrade if 'concurrent' is true.)
+ */
+static void
+rebuild_relation(Relation OldHeap, Relation index, bool verbose,
+ Oid ident_idx)
{
Oid tableOid = RelationGetRelid(OldHeap);
Oid accessMethod = OldHeap->rd_rel->relam;
@@ -644,13 +963,55 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose)
Oid OIDNewHeap;
Relation NewHeap;
char relpersistence;
- bool is_system_catalog;
bool swap_toast_by_content;
TransactionId frozenXid;
MultiXactId cutoffMulti;
+ bool concurrent = OidIsValid(ident_idx);
+ Snapshot snapshot = NULL;
+#if USE_ASSERT_CHECKING
+ LOCKMODE lmode;
- Assert(CheckRelationLockedByMe(OldHeap, AccessExclusiveLock, false) &&
- (index == NULL || CheckRelationLockedByMe(index, AccessExclusiveLock, false)));
+ lmode = concurrent ? ShareUpdateExclusiveLock : AccessExclusiveLock;
+
+ Assert(CheckRelationLockedByMe(OldHeap, lmode, false));
+ Assert(index == NULL || CheckRelationLockedByMe(index, lmode, false));
+#endif
+
+ if (concurrent)
+ {
+ /*
+ * The worker needs to be member of the locking group we're the leader
+ * of. We ought to become the leader before the worker starts. The
+ * worker will join the group as soon as it starts.
+ *
+ * This is to make sure that the deadlock described below is
+ * detectable by deadlock.c: if the worker waits for a transaction to
+ * complete and we are waiting for the worker output, then effectively
+ * we (i.e. this backend) are waiting for that transaction.
+ */
+ BecomeLockGroupLeader();
+
+ /*
+ * Start the worker that decodes data changes applied while we're
+ * copying the table contents.
+ *
+ * Note that the worker has to wait for all transactions with XID
+ * already assigned to finish. If some of those transactions is
+ * waiting for a lock conflicting with ShareUpdateExclusiveLock on our
+ * table (e.g. it runs CREATE INDEX), we can end up in a deadlock.
+ * Not sure this risk is worth unlocking/locking the table (and its
+ * clustering index) and checking again if it's still eligible for
+ * REPACK CONCURRENTLY.
+ */
+ start_repack_decoding_worker(tableOid);
+
+ /*
+ * Wait until the worker has the initial snapshot and retrieve it.
+ */
+ snapshot = get_initial_snapshot(decoding_worker);
+
+ PushActiveSnapshot(snapshot);
+ }
/* for CLUSTER or REPACK USING INDEX, mark the index as the one to use */
if (index != NULL)
@@ -658,7 +1019,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose)
/* Remember info about rel before closing OldHeap */
relpersistence = OldHeap->rd_rel->relpersistence;
- is_system_catalog = IsSystemRelation(OldHeap);
/*
* Create the transient table that will receive the re-ordered data.
@@ -674,30 +1034,59 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose)
NewHeap = table_open(OIDNewHeap, NoLock);
/* Copy the heap data into the new table in the desired order */
- copy_table_data(NewHeap, OldHeap, index, verbose,
+ copy_table_data(NewHeap, OldHeap, index, snapshot, verbose,
&swap_toast_by_content, &frozenXid, &cutoffMulti);
+ /* The historic snapshot won't be needed anymore. */
+ if (snapshot)
+ {
+ PopActiveSnapshot();
+ UpdateActiveSnapshotCommandId();
+ }
- /* Close relcache entries, but keep lock until transaction commit */
- table_close(OldHeap, NoLock);
- if (index)
- index_close(index, NoLock);
+ if (concurrent)
+ {
+ Assert(!swap_toast_by_content);
- /*
- * Close the new relation so it can be dropped as soon as the storage is
- * swapped. The relation is not visible to others, so no need to unlock it
- * explicitly.
- */
- table_close(NewHeap, NoLock);
+ /*
+ * Close the index, but keep the lock. Both heaps will be closed by
+ * the following call.
+ */
+ if (index)
+ index_close(index, NoLock);
- /*
- * Swap the physical files of the target and transient tables, then
- * rebuild the target's indexes and throw away the transient table.
- */
- finish_heap_swap(tableOid, OIDNewHeap, is_system_catalog,
- swap_toast_by_content, false, true,
- frozenXid, cutoffMulti,
- relpersistence);
+ rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx,
+ frozenXid, cutoffMulti);
+
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_FINAL_CLEANUP);
+ }
+ else
+ {
+ bool is_system_catalog = IsSystemRelation(OldHeap);
+
+ /* Close relcache entries, but keep lock until transaction commit */
+ table_close(OldHeap, NoLock);
+ if (index)
+ index_close(index, NoLock);
+
+ /*
+ * Close the new relation so it can be dropped as soon as the storage
+ * is swapped. The relation is not visible to others, so no need to
+ * unlock it explicitly.
+ */
+ table_close(NewHeap, NoLock);
+
+ /*
+ * Swap the physical files of the target and transient tables, then
+ * rebuild the target's indexes and throw away the transient table.
+ */
+ finish_heap_swap(tableOid, OIDNewHeap, is_system_catalog,
+ swap_toast_by_content, false, true,
+ true, /* reindex */
+ frozenXid, cutoffMulti,
+ relpersistence);
+ }
}
@@ -832,15 +1221,18 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, Oid NewAccessMethod,
/*
* Do the physical copying of table data.
*
+ * 'snapshot' and 'decoding_ctx': see table_relation_copy_for_cluster(). Pass
+ * iff concurrent processing is required.
+ *
* There are three output parameters:
* *pSwapToastByContent is set true if toast tables must be swapped by content.
* *pFreezeXid receives the TransactionId used as freeze cutoff point.
* *pCutoffMulti receives the MultiXactId used as a cutoff point.
*/
static void
-copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verbose,
- bool *pSwapToastByContent, TransactionId *pFreezeXid,
- MultiXactId *pCutoffMulti)
+copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
+ Snapshot snapshot, bool verbose, bool *pSwapToastByContent,
+ TransactionId *pFreezeXid, MultiXactId *pCutoffMulti)
{
Relation relRelation;
HeapTuple reltup;
@@ -857,6 +1249,10 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verb
int elevel = verbose ? INFO : DEBUG2;
PGRUsage ru0;
char *nspname;
+ bool concurrent = snapshot != NULL;
+ LOCKMODE lmode;
+
+ lmode = concurrent ? ShareUpdateExclusiveLock : AccessExclusiveLock;
pg_rusage_init(&ru0);
@@ -885,7 +1281,7 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verb
* will be held till end of transaction.
*/
if (OldHeap->rd_rel->reltoastrelid)
- LockRelationOid(OldHeap->rd_rel->reltoastrelid, AccessExclusiveLock);
+ LockRelationOid(OldHeap->rd_rel->reltoastrelid, lmode);
/*
* If both tables have TOAST tables, perform toast swap by content. It is
@@ -894,7 +1290,8 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verb
* swap by links. This is okay because swap by content is only essential
* for system catalogs, and we don't support schema changes for them.
*/
- if (OldHeap->rd_rel->reltoastrelid && NewHeap->rd_rel->reltoastrelid)
+ if (OldHeap->rd_rel->reltoastrelid && NewHeap->rd_rel->reltoastrelid &&
+ !concurrent)
{
*pSwapToastByContent = true;
@@ -915,6 +1312,10 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verb
* follow the toast pointers to the wrong place. (It would actually
* work for values copied over from the old toast table, but not for
* any values that we toast which were previously not toasted.)
+ *
+ * This would not work with CONCURRENTLY because we may need to delete
+ * TOASTed tuples from the new heap. With this hack, we'd delete them
+ * from the old heap.
*/
NewHeap->rd_toastoid = OldHeap->rd_rel->reltoastrelid;
}
@@ -990,7 +1391,8 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verb
* values (e.g. because the AM doesn't use freezing).
*/
table_relation_copy_for_cluster(OldHeap, NewHeap, OldIndex, use_sort,
- cutoffs.OldestXmin, &cutoffs.FreezeLimit,
+ cutoffs.OldestXmin, snapshot,
+ &cutoffs.FreezeLimit,
&cutoffs.MultiXactCutoff,
&num_tuples, &tups_vacuumed,
&tups_recently_dead);
@@ -999,7 +1401,11 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, bool verb
*pFreezeXid = cutoffs.FreezeLimit;
*pCutoffMulti = cutoffs.MultiXactCutoff;
- /* Reset rd_toastoid just to be tidy --- it shouldn't be looked at again */
+ /*
+ * Reset rd_toastoid just to be tidy --- it shouldn't be looked at again.
+ * In the CONCURRENTLY case, we need to set it again before applying the
+ * concurrent changes.
+ */
NewHeap->rd_toastoid = InvalidOid;
num_pages = RelationGetNumberOfBlocks(NewHeap);
@@ -1457,14 +1863,13 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
bool swap_toast_by_content,
bool check_constraints,
bool is_internal,
+ bool reindex,
TransactionId frozenXid,
MultiXactId cutoffMulti,
char newrelpersistence)
{
ObjectAddress object;
Oid mapped_tables[4];
- int reindex_flags;
- ReindexParams reindex_params = {0};
int i;
/* Report that we are now swapping relation files */
@@ -1490,39 +1895,47 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
if (is_system_catalog)
CacheInvalidateCatalog(OIDOldHeap);
- /*
- * Rebuild each index on the relation (but not the toast table, which is
- * all-new at this point). It is important to do this before the DROP
- * step because if we are processing a system catalog that will be used
- * during DROP, we want to have its indexes available. There is no
- * advantage to the other order anyway because this is all transactional,
- * so no chance to reclaim disk space before commit. We do not need a
- * final CommandCounterIncrement() because reindex_relation does it.
- *
- * Note: because index_build is called via reindex_relation, it will never
- * set indcheckxmin true for the indexes. This is OK even though in some
- * sense we are building new indexes rather than rebuilding existing ones,
- * because the new heap won't contain any HOT chains at all, let alone
- * broken ones, so it can't be necessary to set indcheckxmin.
- */
- reindex_flags = REINDEX_REL_SUPPRESS_INDEX_USE;
- if (check_constraints)
- reindex_flags |= REINDEX_REL_CHECK_CONSTRAINTS;
+ if (reindex)
+ {
+ int reindex_flags;
+ ReindexParams reindex_params = {0};
- /*
- * Ensure that the indexes have the same persistence as the parent
- * relation.
- */
- if (newrelpersistence == RELPERSISTENCE_UNLOGGED)
- reindex_flags |= REINDEX_REL_FORCE_INDEXES_UNLOGGED;
- else if (newrelpersistence == RELPERSISTENCE_PERMANENT)
- reindex_flags |= REINDEX_REL_FORCE_INDEXES_PERMANENT;
+ /*
+ * Rebuild each index on the relation (but not the toast table, which
+ * is all-new at this point). It is important to do this before the
+ * DROP step because if we are processing a system catalog that will
+ * be used during DROP, we want to have its indexes available. There
+ * is no advantage to the other order anyway because this is all
+ * transactional, so no chance to reclaim disk space before commit. We
+ * do not need a final CommandCounterIncrement() because
+ * reindex_relation does it.
+ *
+ * Note: because index_build is called via reindex_relation, it will
+ * never set indcheckxmin true for the indexes. This is OK even
+ * though in some sense we are building new indexes rather than
+ * rebuilding existing ones, because the new heap won't contain any
+ * HOT chains at all, let alone broken ones, so it can't be necessary
+ * to set indcheckxmin.
+ */
+ reindex_flags = REINDEX_REL_SUPPRESS_INDEX_USE;
+ if (check_constraints)
+ reindex_flags |= REINDEX_REL_CHECK_CONSTRAINTS;
- /* Report that we are now reindexing relations */
- pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
- PROGRESS_REPACK_PHASE_REBUILD_INDEX);
+ /*
+ * Ensure that the indexes have the same persistence as the parent
+ * relation.
+ */
+ if (newrelpersistence == RELPERSISTENCE_UNLOGGED)
+ reindex_flags |= REINDEX_REL_FORCE_INDEXES_UNLOGGED;
+ else if (newrelpersistence == RELPERSISTENCE_PERMANENT)
+ reindex_flags |= REINDEX_REL_FORCE_INDEXES_PERMANENT;
- reindex_relation(NULL, OIDOldHeap, reindex_flags, &reindex_params);
+ /* Report that we are now reindexing relations */
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_REBUILD_INDEX);
+
+ reindex_relation(NULL, OIDOldHeap, reindex_flags, &reindex_params);
+ }
/* Report that we are now doing clean up */
pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
@@ -1566,6 +1979,17 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
object.objectId = OIDNewHeap;
object.objectSubId = 0;
+ if (!reindex)
+ {
+ /*
+ * Make sure the changes in pg_class are visible. This is especially
+ * important if !swap_toast_by_content, so that the correct TOAST
+ * relation is dropped. (reindex_relation() above did not help in this
+ * case))
+ */
+ CommandCounterIncrement();
+ }
+
/*
* The new relation is local to our transaction and we know nothing
* depends on it, so DROP_RESTRICT should be OK.
@@ -1605,7 +2029,7 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
/* Get the associated valid index to be renamed */
toastidx = toast_get_valid_index(newrel->rd_rel->reltoastrelid,
- NoLock);
+ AccessExclusiveLock);
/* rename the toast table ... */
snprintf(NewToastName, NAMEDATALEN, "pg_toast_%u",
@@ -1876,7 +2300,8 @@ repack_is_permitted_for_relation(RepackCommand cmd, Oid relid, Oid userid)
* case, if an index name is given, it's up to the caller to resolve it.
*/
static Relation
-process_single_relation(RepackStmt *stmt, ClusterParams *params)
+process_single_relation(RepackStmt *stmt, LOCKMODE lockmode, bool isTopLevel,
+ ClusterParams *params)
{
Relation rel;
Oid tableOid;
@@ -1893,13 +2318,9 @@ process_single_relation(RepackStmt *stmt, ClusterParams *params)
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("ANALYZE option must be specified when a column list is provided"));
- /*
- * Find, lock, and check permissions on the table. We obtain
- * AccessExclusiveLock right away to avoid lock-upgrade hazard in the
- * single-transaction case.
- */
+ /* Find, lock, and check permissions on the table. */
tableOid = RangeVarGetRelidExtended(stmt->relation->relation,
- AccessExclusiveLock,
+ lockmode,
0,
RangeVarCallbackMaintainsTable,
NULL);
@@ -1924,13 +2345,14 @@ process_single_relation(RepackStmt *stmt, ClusterParams *params)
return rel;
else
{
- Oid indexOid;
+ Oid indexOid = InvalidOid;
indexOid = determine_clustered_index(rel, stmt->usingindex,
stmt->indexname);
if (OidIsValid(indexOid))
- check_index_is_clusterable(rel, indexOid, AccessExclusiveLock);
- cluster_rel(stmt->command, rel, indexOid, params);
+ check_index_is_clusterable(rel, indexOid, lockmode);
+
+ cluster_rel(stmt->command, rel, indexOid, params, isTopLevel);
/*
* Do an analyze, if requested. We close the transaction and start a
@@ -2025,3 +2447,1182 @@ RepackCommandAsString(RepackCommand cmd)
}
return "???"; /* keep compiler quiet */
}
+
+/*
+ * Apply all the changes stored in 'file'.
+ */
+static void
+apply_concurrent_changes(BufFile *file, ChangeContext *chgcxt)
+{
+ ConcurrentChangeKind kind = '\0';
+ Relation rel = chgcxt->cc_rel;
+ TupleTableSlot *spilled_tuple;
+ TupleTableSlot *old_update_tuple;
+ TupleTableSlot *ondisk_tuple;
+ MemoryContext apply_cxt;
+ bool have_old_tuple = false;
+
+ /*
+ * Use a separate memory context for the tuples and any memory needed to
+ * process them.
+ *
+ * XXX would this be better with GenerationContextCreate?
+ */
+ apply_cxt = AllocSetContextCreate(TopTransactionContext,
+ "REPACK - apply",
+ ALLOCSET_DEFAULT_SIZES);
+
+ spilled_tuple = MakeSingleTupleTableSlot(RelationGetDescr(rel),
+ &TTSOpsVirtual);
+ ondisk_tuple = MakeSingleTupleTableSlot(RelationGetDescr(rel),
+ table_slot_callbacks(rel));
+ old_update_tuple = MakeSingleTupleTableSlot(RelationGetDescr(rel),
+ &TTSOpsVirtual);
+
+ while (true)
+ {
+ size_t nread;
+ ConcurrentChangeKind prevkind = kind;
+
+ CHECK_FOR_INTERRUPTS();
+
+ nread = BufFileReadMaybeEOF(file, &kind, 1, true);
+ if (nread == 0) /* done with the file? */
+ break;
+
+ /*
+ * If this is the old tuple for an update, read it into the tuple slot
+ * and go to the next one. The update itself will be executed on the
+ * next iteration, when we receive the NEW tuple.
+ */
+ if (kind == CHANGE_UPDATE_OLD)
+ {
+ restore_tuple(file, rel, old_update_tuple);
+ have_old_tuple = true;
+ continue;
+ }
+
+ /*
+ * Just before an UPDATE or DELETE, we must update the command
+ * counter, because the change could refer to a tuple that we have
+ * just inserted; and before an INSERT, we have to do this also if the
+ * previous command was either update or delete.
+ *
+ * With this approach we don't spend so many CCIs for long strings of
+ * only INSERTs, which can't affect one another.
+ */
+ if (kind == CHANGE_UPDATE_NEW || kind == CHANGE_DELETE ||
+ (kind == CHANGE_INSERT && (prevkind == CHANGE_UPDATE_NEW ||
+ prevkind == CHANGE_DELETE)))
+ {
+ CommandCounterIncrement();
+ UpdateActiveSnapshotCommandId();
+ }
+
+ /*
+ * Now restore the tuple into the slot and execute the change.
+ */
+ restore_tuple(file, rel, spilled_tuple);
+
+ if (kind == CHANGE_INSERT)
+ {
+ apply_concurrent_insert(rel, spilled_tuple, chgcxt);
+ }
+ else if (kind == CHANGE_DELETE)
+ {
+ bool found;
+
+ /* Find the tuple to be deleted */
+ found = find_target_tuple(rel, chgcxt, spilled_tuple, ondisk_tuple);
+ if (!found)
+ elog(ERROR, "failed to find target tuple");
+ apply_concurrent_delete(rel, ondisk_tuple);
+ }
+ else if (kind == CHANGE_UPDATE_NEW)
+ {
+ TupleTableSlot *key;
+ bool found;
+
+ if (have_old_tuple)
+ key = old_update_tuple;
+ else
+ key = spilled_tuple;
+
+ /* Find the tuple to be updated or deleted. */
+ found = find_target_tuple(rel, chgcxt, key, ondisk_tuple);
+ if (!found)
+ elog(ERROR, "failed to find target tuple");
+
+ /*
+ * If 'tup' contains TOAST pointers, they point to the old
+ * relation's toast. Copy the corresponding TOAST pointers for the
+ * new relation from the existing tuple. (The fact that we
+ * received a TOAST pointer here implies that the attribute hasn't
+ * changed.)
+ */
+ adjust_toast_pointers(rel, spilled_tuple, ondisk_tuple);
+
+ apply_concurrent_update(rel, spilled_tuple, ondisk_tuple, chgcxt);
+
+ ExecClearTuple(old_update_tuple);
+ have_old_tuple = false;
+ }
+ else
+ elog(ERROR, "unrecognized kind of change: %d", kind);
+
+ MemoryContextReset(apply_cxt);
+ }
+
+ /* Cleanup. */
+ ExecDropSingleTupleTableSlot(spilled_tuple);
+ ExecDropSingleTupleTableSlot(ondisk_tuple);
+ ExecDropSingleTupleTableSlot(old_update_tuple);
+ MemoryContextDelete(apply_cxt);
+}
+
+/*
+ * Apply an insert from the spill of concurrent changes to the new copy of the
+ * table.
+ */
+static void
+apply_concurrent_insert(Relation rel, TupleTableSlot *slot,
+ ChangeContext *chgcxt)
+{
+ List *recheck;
+
+ /* Put the tuple in the table, but make sure it won't be decoded */
+ table_tuple_insert(rel, slot, GetCurrentCommandId(true),
+ HEAP_INSERT_NO_LOGICAL, NULL);
+
+ /*
+ * Update indexes with this new tuple. Because we're merely replaying an
+ * action that already happened, we have no use for the recheck list of
+ * indexes returned, so just free it. XXX or maybe just leave it?
+ */
+ recheck = ExecInsertIndexTuples(chgcxt->cc_rri,
+ chgcxt->cc_estate,
+ 0,
+ slot,
+ NIL, NULL);
+ list_free(recheck);
+
+ pgstat_progress_incr_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, 1);
+
+ ResetPerTupleExprContext(chgcxt->cc_estate);
+}
+
+/*
+ * Apply an update from the spill of concurrent changes to the new copy of the
+ * table.
+ */
+static void
+apply_concurrent_update(Relation rel, TupleTableSlot *spilled_tuple,
+ TupleTableSlot *ondisk_tuple,
+ ChangeContext *chgcxt)
+{
+ LockTupleMode lockmode;
+ TM_FailureData tmfd;
+ TU_UpdateIndexes update_indexes;
+ TM_Result res;
+ List *recheck;
+
+ /*
+ * Carry out the update, avoiding logical decoding info.
+ */
+ res = table_tuple_update(rel, &(ondisk_tuple->tts_tid), spilled_tuple,
+ GetCurrentCommandId(true),
+ InvalidSnapshot,
+ InvalidSnapshot,
+ TABLE_UPDATE_NO_LOGICAL,
+ &tmfd, &lockmode, &update_indexes);
+ if (res != TM_Ok)
+ ereport(ERROR,
+ errmsg("failed to apply concurrent UPDATE"));
+
+ if (update_indexes != TU_None)
+ {
+ bits32 flags = EIIT_IS_UPDATE;
+
+ if (update_indexes == TU_Summarizing)
+ flags |= EIIT_ONLY_SUMMARIZING;
+ recheck = ExecInsertIndexTuples(chgcxt->cc_rri,
+ chgcxt->cc_estate,
+ flags,
+ spilled_tuple,
+ NIL, NULL);
+ list_free(recheck);
+ ResetPerTupleExprContext(chgcxt->cc_estate);
+ }
+
+ pgstat_progress_incr_param(PROGRESS_REPACK_HEAP_TUPLES_UPDATED, 1);
+}
+
+static void
+apply_concurrent_delete(Relation rel, TupleTableSlot *slot)
+{
+ TM_Result res;
+ TM_FailureData tmfd;
+
+ /*
+ * Delete tuple from the new heap.
+ *
+ * Do it like in simple_heap_delete(), except for 'wal_logical' (and
+ * except for 'wait').
+ */
+ res = table_tuple_delete(rel, &(slot->tts_tid),
+ GetCurrentCommandId(true),
+ InvalidSnapshot, InvalidSnapshot,
+ TABLE_DELETE_NO_LOGICAL,
+ &tmfd);
+
+ if (res != TM_Ok)
+ ereport(ERROR,
+ errmsg("failed to apply concurrent DELETE"));
+
+ pgstat_progress_incr_param(PROGRESS_REPACK_HEAP_TUPLES_DELETED, 1);
+}
+
+/*
+ * Read tuple from file and put it in the input slot.
+ *
+ * External attributes are stored in separate memory chunks, in order to avoid
+ * exceeding MaxAllocSize - that could happen if the individual attributes are
+ * smaller than MaxAllocSize but the whole tuple is bigger.
+ */
+static void
+restore_tuple(BufFile *file, Relation relation, TupleTableSlot *slot)
+{
+ uint32 t_len;
+ HeapTuple tup;
+ MemoryContext oldcxt;
+ int natt_ext;
+
+ oldcxt = MemoryContextSwitchTo(slot->tts_mcxt);
+
+ /* Read the tuple. */
+ BufFileReadExact(file, &t_len, sizeof(t_len));
+ tup = (HeapTuple) palloc(HEAPTUPLESIZE + t_len);
+ tup->t_data = (HeapTupleHeader) ((char *) tup + HEAPTUPLESIZE);
+ BufFileReadExact(file, tup->t_data, t_len);
+ tup->t_len = t_len;
+ ItemPointerSetInvalid(&tup->t_self);
+ tup->t_tableOid = RelationGetRelid(relation);
+
+ /*
+ * Put the tuple we read in a slot. This deforms it, so that we can hack
+ * the external attributes in place.
+ */
+ ExecForceStoreHeapTuple(tup, slot, false);
+
+ /*
+ * Next, read any attributes we stored separately into the tts_values
+ * array elements expecting them, if any. This matches
+ * repack_store_change.
+ */
+ BufFileReadExact(file, &natt_ext, sizeof(natt_ext));
+ if (natt_ext > 0)
+ {
+ TupleDesc desc = slot->tts_tupleDescriptor;
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(desc, i);
+ varlena *varlen;
+ union
+ {
+ alignas(int32) varlena hdr;
+ char data[sizeof(void *)];
+ } chunk_header;
+ void *value;
+ Size varlensz;
+
+ if (attr->attisdropped || attr->attlen != -1)
+ continue;
+ if (slot_attisnull(slot, i + 1))
+ continue;
+ varlen = (varlena *) DatumGetPointer(slot->tts_values[i]);
+ if (!VARATT_IS_EXTERNAL(varlen))
+ continue;
+ slot_getsomeattrs(slot, i + 1);
+
+ BufFileReadExact(file, &chunk_header, VARHDRSZ);
+ varlensz = VARSIZE_ANY(&chunk_header);
+
+ value = palloc(varlensz);
+ SET_VARSIZE(value, VARSIZE_ANY(&chunk_header));
+ BufFileReadExact(file, (char *) value + VARHDRSZ, varlensz - VARHDRSZ);
+
+ slot->tts_values[i] = PointerGetDatum(value);
+ natt_ext--;
+ }
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+}
+
+/*
+ * Adjust 'dest' replacing any EXTERNAL_ONDISK toast pointers with the
+ * corresponding ones from 'src'.
+ */
+static void
+adjust_toast_pointers(Relation relation, TupleTableSlot *dest, TupleTableSlot *src)
+{
+ TupleDesc desc = dest->tts_tupleDescriptor;
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(desc, i);
+ varlena *varlena_dst;
+
+ if (attr->attisdropped)
+ continue;
+ if (attr->attlen != -1)
+ continue;
+ if (slot_attisnull(dest, i + 1))
+ continue;
+
+ slot_getsomeattrs(dest, i + 1);
+
+ varlena_dst = (varlena *) DatumGetPointer(dest->tts_values[i]);
+ if (!VARATT_IS_EXTERNAL_ONDISK(varlena_dst))
+ continue;
+ slot_getsomeattrs(src, i + 1);
+
+ /*
+ * XXX We simply replace the pointer to the Datum from the other one,
+ * which is probably bogus.
+ */
+ dest->tts_values[i] = src->tts_values[i];
+ }
+}
+
+/*
+ * Find the tuple to be updated or deleted by the given data change, whose
+ * tuple has already been loaded into locator.
+ *
+ * If the tuple is found, put it in retrieved and return true. If the tuple is
+ * not found, return false.
+ */
+static bool
+find_target_tuple(Relation rel, ChangeContext *chgcxt, TupleTableSlot *locator,
+ TupleTableSlot *retrieved)
+{
+ Form_pg_index idx = chgcxt->cc_ident_index->rd_index;
+ IndexScanDesc scan;
+ bool retval;
+
+ /*
+ * Scan key is passed by caller, so it does not have to be constructed
+ * multiple times. Key entries have all fields initialized, except for
+ * sk_argument.
+ *
+ * Use the incoming tuple to finalize the scan key.
+ */
+ for (int i = 0; i < chgcxt->cc_ident_key_nentries; i++)
+ {
+ ScanKey entry = &chgcxt->cc_ident_key[i];
+ AttrNumber attno = idx->indkey.values[i];
+
+ entry->sk_argument = locator->tts_values[attno - 1];
+ Assert(!locator->tts_isnull[attno - 1]);
+ }
+
+ /* XXX no instrumentation for now */
+ scan = index_beginscan(rel, chgcxt->cc_ident_index, GetActiveSnapshot(),
+ NULL, chgcxt->cc_ident_key_nentries, 0);
+ index_rescan(scan, chgcxt->cc_ident_key, chgcxt->cc_ident_key_nentries, NULL, 0);
+ retval = index_getnext_slot(scan, ForwardScanDirection, retrieved);
+ index_endscan(scan);
+
+ return retval;
+}
+
+/*
+ * Decode and apply concurrent changes, up to (and including) the record whose
+ * LSN is 'end_of_wal'.
+ *
+ * XXX the names "process_concurrent_changes" and "apply_concurrent_changes"
+ * are far too similar to each other.
+ */
+static void
+process_concurrent_changes(XLogRecPtr end_of_wal, ChangeContext *chgcxt, bool done)
+{
+ DecodingWorkerShared *shared;
+ char fname[MAXPGPATH];
+ BufFile *file;
+
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_CATCH_UP);
+
+ /* Ask the worker for the file. */
+ shared = (DecodingWorkerShared *) dsm_segment_address(decoding_worker->seg);
+ SpinLockAcquire(&shared->mutex);
+ shared->lsn_upto = end_of_wal;
+ shared->done = done;
+ SpinLockRelease(&shared->mutex);
+
+ /*
+ * The worker needs to finish processing of the current WAL record. Even
+ * if it's idle, it'll need to close the output file. Thus we're likely to
+ * wait, so prepare for sleep.
+ */
+ ConditionVariablePrepareToSleep(&shared->cv);
+ for (;;)
+ {
+ int last_exported;
+
+ SpinLockAcquire(&shared->mutex);
+ last_exported = shared->last_exported;
+ SpinLockRelease(&shared->mutex);
+
+ /*
+ * Has the worker exported the file we are waiting for?
+ */
+ if (last_exported == chgcxt->cc_file_seq)
+ break;
+
+ ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT);
+ }
+ ConditionVariableCancelSleep();
+
+ /* Open the file. */
+ DecodingWorkerFileName(fname, shared->relid, chgcxt->cc_file_seq);
+ file = BufFileOpenFileSet(&shared->sfs.fs, fname, O_RDONLY, false);
+ apply_concurrent_changes(file, chgcxt);
+
+ BufFileClose(file);
+
+ /* Get ready for the next file. */
+ chgcxt->cc_file_seq++;
+}
+
+/*
+ * Initialize the ChangeContext struct for the given relation, with
+ * the given index as identity index.
+ */
+static void
+initialize_change_context(ChangeContext *chgcxt,
+ Relation relation, Oid ident_index_id)
+{
+ chgcxt->cc_rel = relation;
+
+ /* Only initialize fields needed by ExecInsertIndexTuples(). */
+ chgcxt->cc_estate = CreateExecutorState();
+
+ chgcxt->cc_rri = (ResultRelInfo *) palloc(sizeof(ResultRelInfo));
+ InitResultRelInfo(chgcxt->cc_rri, relation, 0, 0, 0);
+ ExecOpenIndices(chgcxt->cc_rri, false);
+
+ /*
+ * The table's relcache entry already has the relcache entry for the
+ * identity index; find that.
+ */
+ chgcxt->cc_ident_index = NULL;
+ for (int i = 0; i < chgcxt->cc_rri->ri_NumIndices; i++)
+ {
+ Relation ind_rel;
+
+ ind_rel = chgcxt->cc_rri->ri_IndexRelationDescs[i];
+ if (ind_rel->rd_id == ident_index_id)
+ {
+ chgcxt->cc_ident_index = ind_rel;
+ break;
+ }
+ }
+ if (chgcxt->cc_ident_index == NULL)
+ elog(ERROR, "failed to find identity index");
+
+ /* Set up for scanning said identity index */
+ {
+ Form_pg_index indexForm;
+
+ indexForm = chgcxt->cc_ident_index->rd_index;
+ chgcxt->cc_ident_key_nentries = indexForm->indnkeyatts;
+ chgcxt->cc_ident_key = (ScanKey) palloc_array(ScanKeyData, indexForm->indnkeyatts);
+ for (int i = 0; i < indexForm->indnkeyatts; i++)
+ {
+ ScanKey entry;
+ Oid opfamily,
+ opcintype,
+ opno,
+ opcode;
+
+ entry = &chgcxt->cc_ident_key[i];
+
+ opfamily = chgcxt->cc_ident_index->rd_opfamily[i];
+ opcintype = chgcxt->cc_ident_index->rd_opcintype[i];
+ opno = get_opfamily_member(opfamily, opcintype, opcintype,
+ BTEqualStrategyNumber);
+ if (!OidIsValid(opno))
+ elog(ERROR, "failed to find = operator for type %u", opcintype);
+ opcode = get_opcode(opno);
+ if (!OidIsValid(opcode))
+ elog(ERROR, "failed to find = operator for operator %u", opno);
+
+ /* Initialize everything but argument. */
+ ScanKeyInit(entry,
+ i + 1,
+ BTEqualStrategyNumber, opcode,
+ (Datum) NULL);
+ entry->sk_collation = chgcxt->cc_ident_index->rd_indcollation[i];
+ }
+ }
+
+ chgcxt->cc_file_seq = WORKER_FILE_SNAPSHOT + 1;
+}
+
+/*
+ * Free up resources taken by a ChangeContext.
+ */
+static void
+release_change_context(ChangeContext *chgcxt)
+{
+ ExecCloseIndices(chgcxt->cc_rri);
+ FreeExecutorState(chgcxt->cc_estate);
+ /* XXX are these pfrees necessary? */
+ pfree(chgcxt->cc_rri);
+ pfree(chgcxt->cc_ident_key);
+}
+
+/*
+ * The final steps of rebuild_relation() for concurrent processing.
+ *
+ * On entry, NewHeap is locked in AccessExclusiveLock mode. OldHeap and its
+ * clustering index (if one is passed) are still locked in a mode that allows
+ * concurrent data changes. On exit, both tables and their indexes are closed,
+ * but locked in AccessExclusiveLock mode.
+ */
+static void
+rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
+ Oid identIdx, TransactionId frozenXid,
+ MultiXactId cutoffMulti)
+{
+ LOCKMODE lockmode_old PG_USED_FOR_ASSERTS_ONLY;
+ List *ind_oids_new;
+ Oid old_table_oid = RelationGetRelid(OldHeap);
+ Oid new_table_oid = RelationGetRelid(NewHeap);
+ List *ind_oids_old = RelationGetIndexList(OldHeap);
+ ListCell *lc,
+ *lc2;
+ char relpersistence;
+ bool is_system_catalog;
+ Oid ident_idx_new;
+ XLogRecPtr wal_insert_ptr,
+ end_of_wal;
+ char dummy_rec_data = '\0';
+ Relation *ind_refs,
+ *ind_refs_p;
+ int nind;
+ ChangeContext chgcxt;
+
+ /* Like in cluster_rel(). */
+ lockmode_old = ShareUpdateExclusiveLock;
+ Assert(CheckRelationLockedByMe(OldHeap, lockmode_old, false));
+ /* This is expected from the caller. */
+ Assert(CheckRelationLockedByMe(NewHeap, AccessExclusiveLock, false));
+
+ /*
+ * Unlike the exclusive case, we build new indexes for the new relation
+ * rather than swapping the storage and reindexing the old relation. The
+ * point is that the index build can take some time, so we do it before we
+ * get AccessExclusiveLock on the old heap and therefore we cannot swap
+ * the heap storage yet.
+ *
+ * index_create() will lock the new indexes using AccessExclusiveLock - no
+ * need to change that. At the same time, we use ShareUpdateExclusiveLock
+ * to lock the existing indexes - that should be enough to prevent others
+ * from changing them while we're repacking the relation. The lock on
+ * table should prevent others from changing the index column list, but
+ * might not be enough for commands like ALTER INDEX ... SET ... (Those
+ * are not necessarily dangerous, but can make user confused if the
+ * changes they do get lost due to REPACK.)
+ */
+ ind_oids_new = build_new_indexes(NewHeap, OldHeap, ind_oids_old);
+
+ /* Find "identity index" on the new relation. */
+ ident_idx_new = InvalidOid;
+ forboth(lc, ind_oids_old, lc2, ind_oids_new)
+ {
+ Oid ind_old = lfirst_oid(lc);
+ Oid ind_new = lfirst_oid(lc2);
+
+ if (identIdx == ind_old)
+ {
+ ident_idx_new = ind_new;
+ break;
+ }
+ }
+
+ /* Should not happen, given our lock on the old relation. */
+ if (!OidIsValid(ident_idx_new))
+ ereport(ERROR,
+ errmsg("identity index missing on the new relation"));
+
+ /* Gather information to apply concurrent changes. */
+ initialize_change_context(&chgcxt, NewHeap, ident_idx_new);
+
+ /*
+ * During testing, wait for another backend to perform concurrent data
+ * changes which we will process below.
+ */
+ INJECTION_POINT("repack-concurrently-before-lock", NULL);
+
+ /*
+ * Flush all WAL records inserted so far (possibly except for the last
+ * incomplete page; see GetInsertRecPtr), to minimize the amount of data
+ * we need to flush while holding exclusive lock on the source table.
+ */
+ wal_insert_ptr = GetInsertRecPtr();
+ XLogFlush(wal_insert_ptr);
+ end_of_wal = GetFlushRecPtr(NULL);
+
+ /*
+ * Apply concurrent changes first time, to minimize the time we need to
+ * hold AccessExclusiveLock. (Quite some amount of WAL could have been
+ * written during the data copying and index creation.)
+ */
+ process_concurrent_changes(end_of_wal, &chgcxt, false);
+
+ /*
+ * Acquire AccessExclusiveLock on the table, its TOAST relation (if there
+ * is one), all its indexes, so that we can swap the files.
+ */
+ LockRelationOid(old_table_oid, AccessExclusiveLock);
+
+ /*
+ * Lock all indexes now, not only the clustering one: all indexes need to
+ * have their files swapped. While doing that, store their relation
+ * references in an array, to handle predicate locks below.
+ */
+ ind_refs_p = ind_refs = palloc_array(Relation, list_length(ind_oids_old));
+ nind = 0;
+ foreach_oid(ind_oid, ind_oids_old)
+ {
+ Relation index;
+
+ index = index_open(ind_oid, AccessExclusiveLock);
+
+ /*
+ * TODO 1) Do we need to check if ALTER INDEX was executed since the
+ * new index was created in build_new_indexes()? 2) Specifically for
+ * the clustering index, should check_index_is_clusterable() be called
+ * here? (Not sure about the latter: ShareUpdateExclusiveLock on the
+ * table probably blocks all commands that affect the result of
+ * check_index_is_clusterable().)
+ */
+ *ind_refs_p = index;
+ ind_refs_p++;
+ nind++;
+ }
+
+ /*
+ * Lock the OldHeap's TOAST relation exclusively - again, the lock is
+ * needed to swap the files.
+ */
+ if (OidIsValid(OldHeap->rd_rel->reltoastrelid))
+ LockRelationOid(OldHeap->rd_rel->reltoastrelid, AccessExclusiveLock);
+
+ /*
+ * Tuples and pages of the old heap will be gone, but the heap will stay.
+ */
+ TransferPredicateLocksToHeapRelation(OldHeap);
+ /* The same for indexes. */
+ for (int i = 0; i < nind; i++)
+ {
+ Relation index = ind_refs[i];
+
+ TransferPredicateLocksToHeapRelation(index);
+
+ /*
+ * References to indexes on the old relation are not needed anymore,
+ * however locks stay till the end of the transaction.
+ */
+ index_close(index, NoLock);
+ }
+ pfree(ind_refs);
+
+ /*
+ * Flush anything we see in WAL, to make sure that all changes committed
+ * while we were waiting for the exclusive lock are available for
+ * decoding. This should not be necessary if all backends had
+ * synchronous_commit set, but we can't rely on this setting.
+ *
+ * Unfortunately, GetInsertRecPtr() may lag behind the actual insert
+ * position, and GetLastImportantRecPtr() points at the start of the last
+ * record rather than at the end. Thus the simplest way to determine the
+ * insert position is to insert a dummy record and use its LSN.
+ *
+ * XXX Consider using GetLastImportantRecPtr() and adding the size of the
+ * last record (plus the total size of all the page headers the record
+ * spans)?
+ */
+ XLogBeginInsert();
+ XLogRegisterData(&dummy_rec_data, 1);
+ wal_insert_ptr = XLogInsert(RM_XLOG_ID, XLOG_NOOP);
+ XLogFlush(wal_insert_ptr);
+ end_of_wal = GetFlushRecPtr(NULL);
+
+ /*
+ * Apply the concurrent changes again. Indicate that the decoding worker
+ * won't be needed anymore.
+ */
+ process_concurrent_changes(end_of_wal, &chgcxt, true);
+
+ /* Remember info about rel before closing OldHeap */
+ relpersistence = OldHeap->rd_rel->relpersistence;
+ is_system_catalog = IsSystemRelation(OldHeap);
+
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_SWAP_REL_FILES);
+
+ /*
+ * Even ShareUpdateExclusiveLock should have prevented others from
+ * creating / dropping indexes (even using the CONCURRENTLY option), so we
+ * do not need to check whether the lists match.
+ */
+ forboth(lc, ind_oids_old, lc2, ind_oids_new)
+ {
+ Oid ind_old = lfirst_oid(lc);
+ Oid ind_new = lfirst_oid(lc2);
+ Oid mapped_tables[4];
+
+ /* Zero out possible results from swapped_relation_files */
+ memset(mapped_tables, 0, sizeof(mapped_tables));
+
+ swap_relation_files(ind_old, ind_new,
+ (old_table_oid == RelationRelationId),
+ false, /* swap_toast_by_content */
+ true,
+ InvalidTransactionId,
+ InvalidMultiXactId,
+ mapped_tables);
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Concurrent processing is not supported for system relations, so
+ * there should be no mapped tables.
+ */
+ for (int i = 0; i < 4; i++)
+ Assert(mapped_tables[i] == 0);
+#endif
+ }
+
+ /* The new indexes must be visible for deletion. */
+ CommandCounterIncrement();
+
+ /* Close the old heap but keep lock until transaction commit. */
+ table_close(OldHeap, NoLock);
+ /* Close the new heap. (We didn't have to open its indexes). */
+ table_close(NewHeap, NoLock);
+
+ /* Cleanup what we don't need anymore. (And close the identity index.) */
+ release_change_context(&chgcxt);
+
+ /*
+ * Swap the relations and their TOAST relations and TOAST indexes. This
+ * also drops the new relation and its indexes.
+ *
+ * (System catalogs are currently not supported.)
+ */
+ Assert(!is_system_catalog);
+ finish_heap_swap(old_table_oid, new_table_oid,
+ is_system_catalog,
+ false, /* swap_toast_by_content */
+ false,
+ true,
+ false, /* reindex */
+ frozenXid, cutoffMulti,
+ relpersistence);
+}
+
+/*
+ * Build indexes on NewHeap according to those on OldHeap.
+ *
+ * OldIndexes is the list of index OIDs on OldHeap. The contained indexes end
+ * up locked using ShareUpdateExclusiveLock.
+ *
+ * A list of OIDs of the corresponding indexes created on NewHeap is
+ * returned. The order of items does match, so we can use these arrays to swap
+ * index storage.
+ */
+static List *
+build_new_indexes(Relation NewHeap, Relation OldHeap, List *OldIndexes)
+{
+ List *result = NIL;
+
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_REBUILD_INDEX);
+
+ foreach_oid(ind_oid, OldIndexes)
+ {
+ Oid ind_oid_new;
+ char *newName;
+ Relation ind;
+
+ ind = index_open(ind_oid, ShareUpdateExclusiveLock);
+
+ newName = ChooseRelationName(get_rel_name(ind_oid),
+ NULL,
+ "repacknew",
+ get_rel_namespace(ind->rd_index->indrelid),
+ false);
+ ind_oid_new = index_create_copy(NewHeap, false, ind_oid,
+ ind->rd_rel->reltablespace, newName);
+ result = lappend_oid(result, ind_oid_new);
+
+ index_close(ind, NoLock);
+ }
+
+ return result;
+}
+
+/*
+ * Try to start a background worker to perform logical decoding of data
+ * changes applied to relation while REPACK CONCURRENTLY is copying its
+ * contents to a new table.
+ */
+static void
+start_repack_decoding_worker(Oid relid)
+{
+ Size size;
+ dsm_segment *seg;
+ DecodingWorkerShared *shared;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+ BackgroundWorker bgw;
+
+ /* Setup shared memory. */
+ size = BUFFERALIGN(offsetof(DecodingWorkerShared, error_queue)) +
+ BUFFERALIGN(REPACK_ERROR_QUEUE_SIZE);
+ seg = dsm_create(size, 0);
+ shared = (DecodingWorkerShared *) dsm_segment_address(seg);
+ shared->lsn_upto = InvalidXLogRecPtr;
+ shared->done = false;
+ SharedFileSetInit(&shared->sfs, seg);
+ shared->last_exported = -1;
+ SpinLockInit(&shared->mutex);
+ shared->dbid = MyDatabaseId;
+
+ /*
+ * This is the UserId set in cluster_rel(). Security context shouldn't be
+ * needed for decoding worker.
+ */
+ shared->roleid = GetUserId();
+ shared->relid = relid;
+ ConditionVariableInit(&shared->cv);
+ shared->backend_proc = MyProc;
+ shared->backend_pid = MyProcPid;
+ shared->backend_proc_number = MyProcNumber;
+
+ mq = shm_mq_create((char *) BUFFERALIGN(shared->error_queue),
+ REPACK_ERROR_QUEUE_SIZE);
+ shm_mq_set_receiver(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+
+ memset(&bgw, 0, sizeof(bgw));
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "REPACK decoding worker for relation \"%s\"",
+ get_rel_name(relid));
+ snprintf(bgw.bgw_type, BGW_MAXLEN, "REPACK decoding worker");
+ bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_restart_time = BGW_NEVER_RESTART;
+ snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "RepackWorkerMain");
+ bgw.bgw_main_arg = UInt32GetDatum(dsm_segment_handle(seg));
+ bgw.bgw_notify_pid = MyProcPid;
+
+ decoding_worker = palloc0_object(DecodingWorker);
+ if (!RegisterDynamicBackgroundWorker(&bgw, &decoding_worker->handle))
+ ereport(ERROR,
+ errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg("out of background worker slots"),
+ errhint("You might need to increase \"%s\".", "max_worker_processes"));
+
+ decoding_worker->seg = seg;
+ decoding_worker->error_mqh = mqh;
+
+ /*
+ * The decoding setup must be done before the caller can have XID assigned
+ * for any reason, otherwise the worker might end up in a deadlock,
+ * waiting for the caller's transaction to end. Therefore wait here until
+ * the worker indicates that it has the logical decoding initialized.
+ */
+ ConditionVariablePrepareToSleep(&shared->cv);
+ for (;;)
+ {
+ bool initialized;
+
+ SpinLockAcquire(&shared->mutex);
+ initialized = shared->initialized;
+ SpinLockRelease(&shared->mutex);
+
+ if (initialized)
+ break;
+
+ ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT);
+ }
+ ConditionVariableCancelSleep();
+}
+
+/*
+ * Stop the decoding worker and cleanup the related resources.
+ *
+ * The worker stops on its own when it knows there is no more work to do, but
+ * we need to stop it explicitly at least on ERROR in the launching backend.
+ */
+static void
+stop_repack_decoding_worker(void)
+{
+ BgwHandleStatus status;
+
+ /* Haven't reached the worker startup? */
+ if (decoding_worker == NULL)
+ return;
+
+ /* Could not register the worker? */
+ if (decoding_worker->handle == NULL)
+ return;
+
+ TerminateBackgroundWorker(decoding_worker->handle);
+ /* The worker should really exit before the REPACK command does. */
+ HOLD_INTERRUPTS();
+ status = WaitForBackgroundWorkerShutdown(decoding_worker->handle);
+ RESUME_INTERRUPTS();
+
+ if (status == BGWH_POSTMASTER_DIED)
+ ereport(FATAL,
+ errcode(ERRCODE_ADMIN_SHUTDOWN),
+ errmsg("postmaster exited during REPACK command"));
+
+ shm_mq_detach(decoding_worker->error_mqh);
+
+ /*
+ * If we could not cancel the current sleep due to ERROR, do that before
+ * we detach from the shared memory the condition variable is located in.
+ * If we did not, the bgworker ERROR handling code would try and fail
+ * badly.
+ */
+ ConditionVariableCancelSleep();
+
+ dsm_detach(decoding_worker->seg);
+ pfree(decoding_worker);
+ decoding_worker = NULL;
+}
+
+/*
+ * Get the initial snapshot from the decoding worker.
+ */
+static Snapshot
+get_initial_snapshot(DecodingWorker *worker)
+{
+ DecodingWorkerShared *shared;
+ char fname[MAXPGPATH];
+ BufFile *file;
+ Size snap_size;
+ char *snap_space;
+ Snapshot snapshot;
+
+ shared = (DecodingWorkerShared *) dsm_segment_address(worker->seg);
+
+ /*
+ * The worker needs to initialize the logical decoding, which usually
+ * takes some time. Therefore it makes sense to prepare for the sleep
+ * first.
+ */
+ ConditionVariablePrepareToSleep(&shared->cv);
+ for (;;)
+ {
+ int last_exported;
+
+ SpinLockAcquire(&shared->mutex);
+ last_exported = shared->last_exported;
+ SpinLockRelease(&shared->mutex);
+
+ /*
+ * Has the worker exported the file we are waiting for?
+ */
+ if (last_exported == WORKER_FILE_SNAPSHOT)
+ break;
+
+ ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT);
+ }
+ ConditionVariableCancelSleep();
+
+ /* Read the snapshot from a file. */
+ DecodingWorkerFileName(fname, shared->relid, WORKER_FILE_SNAPSHOT);
+ file = BufFileOpenFileSet(&shared->sfs.fs, fname, O_RDONLY, false);
+ BufFileReadExact(file, &snap_size, sizeof(snap_size));
+ snap_space = (char *) palloc(snap_size);
+ BufFileReadExact(file, snap_space, snap_size);
+ BufFileClose(file);
+
+ /* Restore it. */
+ snapshot = RestoreSnapshot(snap_space);
+ pfree(snap_space);
+
+ return snapshot;
+}
+
+/*
+ * Generate worker's file name into 'fname', which must be of size MAXPGPATH.
+ * If relations of the same 'relid' happen to be processed at the same time,
+ * they must be from different databases and therefore different backends must
+ * be involved.
+ */
+void
+DecodingWorkerFileName(char *fname, Oid relid, uint32 seq)
+{
+ /* The PID is already present in the fileset name, so we needn't add it */
+ snprintf(fname, MAXPGPATH, "%u-%u", relid, seq);
+}
+
+/*
+ * Handle receipt of an interrupt indicating a repack worker message.
+ *
+ * Note: this is called within a signal handler! All we can do is set
+ * a flag that will cause the next CHECK_FOR_INTERRUPTS() to invoke
+ * ProcessRepackMessages().
+ */
+void
+HandleRepackMessageInterrupt(void)
+{
+ InterruptPending = true;
+ RepackMessagePending = true;
+ SetLatch(MyLatch);
+}
+
+/*
+ * Process any queued protocol messages received from the repack worker.
+ */
+void
+ProcessRepackMessages(void)
+{
+ MemoryContext oldcontext;
+ static MemoryContext hpm_context = NULL;
+
+ /*
+ * Nothing to do if we haven't launched the worker yet or have already
+ * terminated it.
+ */
+ if (decoding_worker == NULL)
+ return;
+
+ /*
+ * This is invoked from ProcessInterrupts(), and since some of the
+ * functions it calls contain CHECK_FOR_INTERRUPTS(), there is a potential
+ * for recursive calls if more signals are received while this runs. It's
+ * unclear that recursive entry would be safe, and it doesn't seem useful
+ * even if it is safe, so let's block interrupts until done.
+ */
+ HOLD_INTERRUPTS();
+
+ /*
+ * Moreover, CurrentMemoryContext might be pointing almost anywhere. We
+ * don't want to risk leaking data into long-lived contexts, so let's do
+ * our work here in a private context that we can reset on each use.
+ */
+ if (hpm_context == NULL) /* first time through? */
+ hpm_context = AllocSetContextCreate(TopMemoryContext,
+ "ProcessRepackMessages",
+ ALLOCSET_DEFAULT_SIZES);
+ else
+ MemoryContextReset(hpm_context);
+
+ oldcontext = MemoryContextSwitchTo(hpm_context);
+
+ /* OK to process messages. Reset the flag saying there are more to do. */
+ RepackMessagePending = false;
+
+ /*
+ * Read as many messages as we can from the worker, but stop when no more
+ * messages can be read from the worker without blocking.
+ */
+ while (true)
+ {
+ shm_mq_result res;
+ Size nbytes;
+ void *data;
+
+ res = shm_mq_receive(decoding_worker->error_mqh, &nbytes,
+ &data, true);
+ if (res == SHM_MQ_WOULD_BLOCK)
+ break;
+ else if (res == SHM_MQ_SUCCESS)
+ {
+ StringInfoData msg;
+
+ initStringInfo(&msg);
+ appendBinaryStringInfo(&msg, data, nbytes);
+ ProcessRepackMessage(&msg);
+ pfree(msg.data);
+ }
+ else
+ {
+ /*
+ * The decoding worker is special in that it exits as soon as it
+ * has its work done. Thus the DETACHED result code is fine.
+ */
+ Assert(res == SHM_MQ_DETACHED);
+
+ break;
+ }
+ }
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Might as well clear the context on our way out */
+ MemoryContextReset(hpm_context);
+
+ RESUME_INTERRUPTS();
+}
+
+/*
+ * Process a single protocol message received from a single parallel worker.
+ */
+static void
+ProcessRepackMessage(StringInfo msg)
+{
+ char msgtype;
+
+ msgtype = pq_getmsgbyte(msg);
+
+ switch (msgtype)
+ {
+ case PqMsg_ErrorResponse:
+ case PqMsg_NoticeResponse:
+ {
+ ErrorData edata;
+
+ /* Parse ErrorResponse or NoticeResponse. */
+ pq_parse_errornotice(msg, &edata);
+
+ /* Death of a worker isn't enough justification for suicide. */
+ edata.elevel = Min(edata.elevel, ERROR);
+
+ /*
+ * Add a context line to show that this is a message propagated
+ * from the worker. Otherwise, it can sometimes be confusing
+ * to understand what actually happened.
+ */
+ if (edata.context)
+ edata.context = psprintf("%s\n%s", edata.context,
+ _("REPACK decoding worker"));
+ else
+ edata.context = pstrdup(_("REPACK decoding worker"));
+
+ /* Rethrow error or print notice. */
+ ThrowErrorData(&edata);
+
+ break;
+ }
+
+ default:
+ {
+ elog(ERROR, "unrecognized message type received from decoding worker: %c (message length %d bytes)",
+ msgtype, msg->len);
+ }
+ }
+}
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 81a55a33ef2..539969d6eef 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -893,6 +893,7 @@ static void
refresh_by_heap_swap(Oid matviewOid, Oid OIDNewHeap, char relpersistence)
{
finish_heap_swap(matviewOid, OIDNewHeap, false, false, true, true,
+ true, /* reindex */
RecentXmin, ReadNextMultiXactId(), relpersistence);
}
diff --git a/src/backend/commands/meson.build b/src/backend/commands/meson.build
index 90c7e37a429..4a0e6f0002b 100644
--- a/src/backend/commands/meson.build
+++ b/src/backend/commands/meson.build
@@ -39,6 +39,7 @@ backend_sources += files(
'proclang.c',
'propgraphcmds.c',
'publicationcmds.c',
+ 'repack_worker.c',
'schemacmds.c',
'seclabel.c',
'sequence.c',
diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
new file mode 100644
index 00000000000..35ab1a25c42
--- /dev/null
+++ b/src/backend/commands/repack_worker.c
@@ -0,0 +1,546 @@
+/*-------------------------------------------------------------------------
+ *
+ * repack_worker.c
+ * Implementation of the background worker for ad-hoc logical decoding
+ * during REPACK (CONCURRENTLY).
+ *
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994-5, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/repack_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/table.h"
+#include "access/xlog_internal.h"
+#include "access/xlogutils.h"
+#include "access/xlogwait.h"
+#include "commands/cluster.h"
+#include "commands/repack_internal.h"
+#include "libpq/pqmq.h"
+#include "replication/snapbuild.h"
+#include "storage/ipc.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
+
+#define REPL_PLUGIN_NAME "pgoutput_repack"
+
+static void repack_worker_internal(dsm_segment *seg);
+static void RepackWorkerShutdown(int code, Datum arg);
+static LogicalDecodingContext *repack_setup_logical_decoding(Oid relid);
+static void repack_cleanup_logical_decoding(LogicalDecodingContext *ctx);
+static void export_initial_snapshot(Snapshot snapshot,
+ DecodingWorkerShared *shared);
+static bool decode_concurrent_changes(LogicalDecodingContext *ctx,
+ DecodingWorkerShared *shared);
+
+
+/* Is this process a REPACK worker? */
+static bool is_repack_worker = false;
+
+/* The WAL segment being decoded. */
+static XLogSegNo repack_current_segment = 0;
+
+/*
+ * Keep track of the table we're processing, to skip logical decoding of data
+ * from other relations.
+ */
+static RelFileLocator repacked_rel_locator = {.relNumber = InvalidOid};
+static RelFileLocator repacked_rel_toast_locator = {.relNumber = InvalidOid};
+
+
+/* REPACK decoding worker entry point */
+void
+RepackWorkerMain(Datum main_arg)
+{
+ dsm_segment *seg;
+ DecodingWorkerShared *shared;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+
+ is_repack_worker = true;
+
+ /*
+ * Override the default bgworker_die() with die() so we can use
+ * CHECK_FOR_INTERRUPTS().
+ */
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ seg = dsm_attach(DatumGetUInt32(main_arg));
+ if (seg == NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not map dynamic shared memory segment"));
+
+ shared = (DecodingWorkerShared *) dsm_segment_address(seg);
+
+ /* Arrange to signal the leader if we exit. */
+ backend_pid = shared->backend_pid;
+ backend_proc_number = shared->backend_proc_number;
+ before_shmem_exit(RepackWorkerShutdown, PointerGetDatum(seg));
+
+ /*
+ * Join locking group - see the comments around the call of
+ * start_repack_decoding_worker().
+ */
+ if (!BecomeLockGroupMember(shared->backend_proc, backend_pid))
+ /* The leader is not running anymore. */
+ return;
+
+ /*
+ * Setup a queue to send error messages to the backend that launched this
+ * worker.
+ */
+ mq = (shm_mq *) (char *) BUFFERALIGN(shared->error_queue);
+ shm_mq_set_sender(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+ pq_redirect_to_shm_mq(seg, mqh);
+ pq_set_parallel_leader(shared->backend_pid,
+ shared->backend_proc_number);
+
+ /* Connect to the database. */
+ BackgroundWorkerInitializeConnectionByOid(shared->dbid, shared->roleid, 0);
+
+ repack_worker_internal(seg);
+}
+
+static void
+repack_worker_internal(dsm_segment *seg)
+{
+ DecodingWorkerShared *shared;
+ LogicalDecodingContext *decoding_ctx;
+ SharedFileSet *sfs;
+ Snapshot snapshot;
+
+ /*
+ * Transaction is needed to open relation, and it also provides us with a
+ * resource owner.
+ */
+ StartTransactionCommand();
+
+ shared = (DecodingWorkerShared *) dsm_segment_address(seg);
+
+ /*
+ * Not sure the spinlock is needed here - the backend should not change
+ * anything in the shared memory until we have serialized the snapshot.
+ */
+ SpinLockAcquire(&shared->mutex);
+ Assert(!XLogRecPtrIsValid(shared->lsn_upto));
+ sfs = &shared->sfs;
+ SpinLockRelease(&shared->mutex);
+
+ SharedFileSetAttach(sfs, seg);
+
+ /*
+ * Prepare to capture the concurrent data changes ourselves.
+ */
+ decoding_ctx = repack_setup_logical_decoding(shared->relid);
+
+ /* Announce that we're ready. */
+ SpinLockAcquire(&shared->mutex);
+ shared->initialized = true;
+ SpinLockRelease(&shared->mutex);
+ ConditionVariableSignal(&shared->cv);
+
+ /* Build the initial snapshot and export it. */
+ snapshot = SnapBuildInitialSnapshot(decoding_ctx->snapshot_builder, true);
+ export_initial_snapshot(snapshot, shared);
+
+ /*
+ * Only historic snapshots should be used now. Do not let us restrict the
+ * progress of xmin horizon.
+ */
+ InvalidateCatalogSnapshot();
+
+ for (;;)
+ {
+ bool stop = decode_concurrent_changes(decoding_ctx, shared);
+
+ if (stop)
+ break;
+
+ }
+
+ /* Cleanup. */
+ repack_cleanup_logical_decoding(decoding_ctx);
+ CommitTransactionCommand();
+}
+
+/*
+ * See ParallelWorkerShutdown for details.
+ */
+static void
+RepackWorkerShutdown(int code, Datum arg)
+{
+ SendProcSignal(backend_pid,
+ PROCSIG_REPACK_MESSAGE,
+ backend_proc_number);
+
+ dsm_detach((dsm_segment *) DatumGetPointer(arg));
+}
+
+bool
+IsRepackWorker(void)
+{
+ return is_repack_worker;
+}
+
+/*
+ * This function is much like pg_create_logical_replication_slot() except that
+ * the new slot is neither released (if anyone else could read changes from
+ * our slot, we could miss changes other backends do while we copy the
+ * existing data into temporary table), nor persisted (it's easier to handle
+ * crash by restarting all the work from scratch).
+ */
+static LogicalDecodingContext *
+repack_setup_logical_decoding(Oid relid)
+{
+ Relation rel;
+ Oid toastrelid;
+ LogicalDecodingContext *ctx;
+ NameData slotname;
+ RepackDecodingState *dstate;
+ MemoryContext oldcxt;
+
+ /*
+ * REPACK CONCURRENTLY is not allowed in a transaction block, so this
+ * should never fire.
+ */
+ Assert(!TransactionIdIsValid(GetTopTransactionIdIfAny()));
+
+ /*
+ * Make sure we can use logical decoding.
+ */
+ CheckSlotPermissions();
+ CheckLogicalDecodingRequirements();
+
+ /*
+ * A single backend should not execute multiple REPACK commands at a time,
+ * so use PID to make the slot unique.
+ *
+ * RS_TEMPORARY so that the slot gets cleaned up on ERROR.
+ */
+ snprintf(NameStr(slotname), NAMEDATALEN, "repack_%d", MyProcPid);
+ ReplicationSlotCreate(NameStr(slotname), true, RS_TEMPORARY, false, false,
+ false);
+
+ EnsureLogicalDecodingEnabled();
+
+ /*
+ * Neither prepare_write nor do_write callback nor update_progress is
+ * useful for us.
+ */
+ ctx = CreateInitDecodingContext(REPL_PLUGIN_NAME,
+ NIL,
+ true,
+ InvalidXLogRecPtr,
+ XL_ROUTINE(.page_read = read_local_xlog_page,
+ .segment_open = wal_segment_open,
+ .segment_close = wal_segment_close),
+ NULL, NULL, NULL);
+
+ /*
+ * We don't have control on setting fast_forward, so at least check it.
+ */
+ Assert(!ctx->fast_forward);
+
+ /* Avoid logical decoding of other relations. */
+ rel = table_open(relid, AccessShareLock);
+ repacked_rel_locator = rel->rd_locator;
+ toastrelid = rel->rd_rel->reltoastrelid;
+ if (OidIsValid(toastrelid))
+ {
+ Relation toastrel;
+
+ /* Avoid logical decoding of other TOAST relations. */
+ toastrel = table_open(toastrelid, AccessShareLock);
+ repacked_rel_toast_locator = toastrel->rd_locator;
+ table_close(toastrel, AccessShareLock);
+ }
+ table_close(rel, AccessShareLock);
+
+ DecodingContextFindStartpoint(ctx);
+
+ /*
+ * decode_concurrent_changes() needs non-blocking callback.
+ */
+ ctx->reader->routine.page_read = read_local_xlog_page_no_wait;
+
+ /* Some WAL records should have been read. */
+ Assert(ctx->reader->EndRecPtr != InvalidXLogRecPtr);
+
+ /*
+ * Initialize repack_current_segment so that we can notice WAL segment
+ * boundaries.
+ */
+ XLByteToSeg(ctx->reader->EndRecPtr, repack_current_segment,
+ wal_segment_size);
+
+ /* Our private state belongs to the decoding context. */
+ oldcxt = MemoryContextSwitchTo(ctx->context);
+
+ /*
+ * read_local_xlog_page_no_wait() needs to be able to indicate the end of
+ * WAL.
+ */
+ ctx->reader->private_data = palloc0_object(ReadLocalXLogPageNoWaitPrivate);
+ dstate = palloc0_object(RepackDecodingState);
+ MemoryContextSwitchTo(oldcxt);
+
+#ifdef USE_ASSERT_CHECKING
+ dstate->relid = relid;
+#endif
+
+ dstate->change_cxt = AllocSetContextCreate(ctx->context,
+ "REPACK - change",
+ ALLOCSET_DEFAULT_SIZES);
+
+ /* The file will be set as soon as we have it opened. */
+ dstate->file = NULL;
+
+ /*
+ * Memory context and resource owner for long-lived resources.
+ */
+ dstate->worker_cxt = CurrentMemoryContext;
+ dstate->worker_resowner = CurrentResourceOwner;
+
+ ctx->output_writer_private = dstate;
+
+ return ctx;
+}
+
+static void
+repack_cleanup_logical_decoding(LogicalDecodingContext *ctx)
+{
+ RepackDecodingState *dstate;
+
+ dstate = (RepackDecodingState *) ctx->output_writer_private;
+ if (dstate->slot)
+ ExecDropSingleTupleTableSlot(dstate->slot);
+
+ FreeDecodingContext(ctx);
+ ReplicationSlotDropAcquired();
+}
+
+/*
+ * Make snapshot available to the backend that launched the decoding worker.
+ */
+static void
+export_initial_snapshot(Snapshot snapshot, DecodingWorkerShared *shared)
+{
+ char fname[MAXPGPATH];
+ BufFile *file;
+ Size snap_size;
+ char *snap_space;
+
+ snap_size = EstimateSnapshotSpace(snapshot);
+ snap_space = (char *) palloc(snap_size);
+ SerializeSnapshot(snapshot, snap_space);
+ FreeSnapshot(snapshot);
+
+ DecodingWorkerFileName(fname, shared->relid, shared->last_exported + 1);
+ file = BufFileCreateFileSet(&shared->sfs.fs, fname);
+ /* To make restoration easier, write the snapshot size first. */
+ BufFileWrite(file, &snap_size, sizeof(snap_size));
+ BufFileWrite(file, snap_space, snap_size);
+ pfree(snap_space);
+ BufFileClose(file);
+
+ /* Increase the counter to tell the backend that the file is available. */
+ SpinLockAcquire(&shared->mutex);
+ shared->last_exported++;
+ SpinLockRelease(&shared->mutex);
+ ConditionVariableSignal(&shared->cv);
+}
+
+/*
+ * Decode logical changes from the WAL sequence and store them to a file.
+ *
+ * If true is returned, there is no more work for the worker.
+ */
+static bool
+decode_concurrent_changes(LogicalDecodingContext *ctx,
+ DecodingWorkerShared *shared)
+{
+ RepackDecodingState *dstate;
+ XLogRecPtr lsn_upto;
+ bool done;
+ char fname[MAXPGPATH];
+
+ dstate = (RepackDecodingState *) ctx->output_writer_private;
+
+ /* Open the output file. */
+ DecodingWorkerFileName(fname, shared->relid, shared->last_exported + 1);
+ dstate->file = BufFileCreateFileSet(&shared->sfs.fs, fname);
+
+ SpinLockAcquire(&shared->mutex);
+ lsn_upto = shared->lsn_upto;
+ done = shared->done;
+ SpinLockRelease(&shared->mutex);
+
+ while (true)
+ {
+ XLogRecord *record;
+ XLogSegNo segno_new;
+ char *errm = NULL;
+ XLogRecPtr end_lsn;
+
+ CHECK_FOR_INTERRUPTS();
+
+ record = XLogReadRecord(ctx->reader, &errm);
+ if (record)
+ {
+ LogicalDecodingProcessRecord(ctx, ctx->reader);
+
+ /*
+ * If WAL segment boundary has been crossed, inform the decoding
+ * system that the catalog_xmin can advance.
+ *
+ * TODO Does it make sense to confirm more often? Segment size
+ * seems appropriate for restart_lsn (because less than a segment
+ * cannot be recycled anyway), however more frequent checks might
+ * be beneficial for catalog_xmin.
+ */
+ end_lsn = ctx->reader->EndRecPtr;
+ XLByteToSeg(end_lsn, segno_new, wal_segment_size);
+ if (segno_new != repack_current_segment)
+ {
+ LogicalConfirmReceivedLocation(end_lsn);
+ elog(DEBUG1, "REPACK: confirmed receive location %X/%X",
+ (uint32) (end_lsn >> 32), (uint32) end_lsn);
+ repack_current_segment = segno_new;
+ }
+ }
+ else
+ {
+ ReadLocalXLogPageNoWaitPrivate *priv;
+
+ if (errm)
+ ereport(ERROR,
+ errmsg("%s", errm));
+
+ /*
+ * In the decoding loop we do not want to get blocked when there
+ * is no more WAL available, otherwise the loop would become
+ * uninterruptible.
+ */
+ priv = (ReadLocalXLogPageNoWaitPrivate *) ctx->reader->private_data;
+ if (priv->end_of_wal)
+ /* Do not miss the end of WAL condition next time. */
+ priv->end_of_wal = false;
+ else
+ ereport(ERROR,
+ errmsg("could not read WAL record"));
+ }
+
+ /*
+ * Whether we could read new record or not, keep checking if
+ * 'lsn_upto' was specified.
+ */
+ if (!XLogRecPtrIsValid(lsn_upto))
+ {
+ SpinLockAcquire(&shared->mutex);
+ lsn_upto = shared->lsn_upto;
+ /* 'done' should be set at the same time as 'lsn_upto' */
+ done = shared->done;
+ SpinLockRelease(&shared->mutex);
+ }
+ if (XLogRecPtrIsValid(lsn_upto) &&
+ ctx->reader->EndRecPtr >= lsn_upto)
+ break;
+
+ if (record == NULL)
+ {
+ int64 timeout = 0;
+ WaitLSNResult res;
+
+ /*
+ * Before we retry reading, wait until new WAL is flushed.
+ *
+ * There is a race condition such that the backend executing
+ * REPACK determines 'lsn_upto', but before it sets the shared
+ * variable, we reach the end of WAL. In that case we'd need to
+ * wait until the next WAL flush (unrelated to REPACK). Although
+ * that should not be a problem in a busy system, it might be
+ * noticeable in other cases, including regression tests (which
+ * are not necessarily executed in parallel). Therefore it makes
+ * sense to use timeout.
+ *
+ * If lsn_upto is valid, WAL records having LSN lower than that
+ * should already have been flushed to disk.
+ */
+ if (!XLogRecPtrIsValid(lsn_upto))
+ timeout = 100L;
+ res = WaitForLSN(WAIT_LSN_TYPE_PRIMARY_FLUSH,
+ ctx->reader->EndRecPtr + 1,
+ timeout);
+ if (res != WAIT_LSN_RESULT_SUCCESS &&
+ res != WAIT_LSN_RESULT_TIMEOUT)
+ ereport(ERROR,
+ errmsg("waiting for WAL failed"));
+ }
+ }
+
+ /*
+ * Close the file so we can make it available to the backend.
+ */
+ BufFileClose(dstate->file);
+ dstate->file = NULL;
+ SpinLockAcquire(&shared->mutex);
+ shared->lsn_upto = InvalidXLogRecPtr;
+ shared->last_exported++;
+ SpinLockRelease(&shared->mutex);
+ ConditionVariableSignal(&shared->cv);
+
+ return done;
+}
+
+/*
+ * Does the WAL record contain a data change that this backend does not need
+ * to decode on behalf of REPACK (CONCURRENTLY)?
+ */
+bool
+change_useless_for_repack(XLogRecordBuffer *buf)
+{
+ XLogReaderState *r = buf->record;
+ RelFileLocator locator;
+
+ /* TOAST locator should not be set unless the main is. */
+ Assert(!OidIsValid(repacked_rel_toast_locator.relNumber) ||
+ OidIsValid(repacked_rel_locator.relNumber));
+
+ /*
+ * Backends not involved in REPACK (CONCURRENTLY) should not do the
+ * filtering.
+ */
+ if (!OidIsValid(repacked_rel_locator.relNumber))
+ return false;
+
+ /*
+ * If the record does not contain the block 0, it's probably not INSERT /
+ * UPDATE / DELETE. In any case, we do not have enough information to
+ * filter the change out.
+ */
+ if (!XLogRecGetBlockTagExtended(r, 0, &locator, NULL, NULL, NULL))
+ return false;
+
+ /*
+ * Decode the change if it belongs to the table we are repacking, or if it
+ * belongs to its TOAST relation.
+ */
+ if (RelFileLocatorEquals(locator, repacked_rel_locator))
+ return false;
+ if (OidIsValid(repacked_rel_toast_locator.relNumber) &&
+ RelFileLocatorEquals(locator, repacked_rel_toast_locator))
+ return false;
+
+ /* Filter out changes of other tables. */
+ return true;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index c69c12dc014..cc70439627b 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -6058,6 +6058,7 @@ ATRewriteTables(AlterTableStmt *parsetree, List **wqueue, LOCKMODE lockmode,
finish_heap_swap(tab->relid, OIDNewHeap,
false, false, true,
!OidIsValid(tab->newTableSpace),
+ true, /* reindex */
RecentXmin,
ReadNextMultiXactId(),
persistence);
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index bce3a2daa24..201835f30a4 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -127,7 +127,7 @@ static void vac_truncate_clog(TransactionId frozenXID,
TransactionId lastSaneFrozenXid,
MultiXactId lastSaneMinMulti);
static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams params,
- BufferAccessStrategy bstrategy);
+ BufferAccessStrategy bstrategy, bool isTopLevel);
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
@@ -630,7 +630,8 @@ vacuum(List *relations, const VacuumParams params, BufferAccessStrategy bstrateg
if (params.options & VACOPT_VACUUM)
{
- if (!vacuum_rel(vrel->oid, vrel->relation, params, bstrategy))
+ if (!vacuum_rel(vrel->oid, vrel->relation, params, bstrategy,
+ isTopLevel))
continue;
}
@@ -2004,7 +2005,7 @@ vac_truncate_clog(TransactionId frozenXID,
*/
static bool
vacuum_rel(Oid relid, RangeVar *relation, VacuumParams params,
- BufferAccessStrategy bstrategy)
+ BufferAccessStrategy bstrategy, bool isTopLevel)
{
LOCKMODE lmode;
Relation rel;
@@ -2295,7 +2296,7 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams params,
/* VACUUM FULL is a variant of REPACK; see cluster.c */
cluster_rel(REPACK_COMMAND_VACUUMFULL, rel, InvalidOid,
- &cluster_params);
+ &cluster_params, isTopLevel);
/* cluster_rel closes the relation, but keeps lock */
rel = NULL;
@@ -2338,7 +2339,8 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams params,
toast_vacuum_params.options |= VACOPT_PROCESS_MAIN;
toast_vacuum_params.toast_parent = relid;
- vacuum_rel(toast_relid, NULL, toast_vacuum_params, bstrategy);
+ vacuum_rel(toast_relid, NULL, toast_vacuum_params, bstrategy,
+ isTopLevel);
}
/*
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 4cd5e262e0f..680c29f35d5 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -1522,14 +1522,17 @@ ExecDeleteAct(ModifyTableContext *context, ResultRelInfo *resultRelInfo,
ItemPointer tupleid, bool changingPart)
{
EState *estate = context->estate;
+ int options = TABLE_DELETE_WAIT;
+
+ if (changingPart)
+ options |= TABLE_DELETE_CHANGING_PART;
return table_tuple_delete(resultRelInfo->ri_RelationDesc, tupleid,
estate->es_output_cid,
estate->es_snapshot,
estate->es_crosscheck_snapshot,
- true /* wait for commit */ ,
- &context->tmfd,
- changingPart);
+ options,
+ &context->tmfd);
}
/*
@@ -2333,7 +2336,7 @@ lreplace:
estate->es_output_cid,
estate->es_snapshot,
estate->es_crosscheck_snapshot,
- true /* wait for commit */ ,
+ TABLE_UPDATE_WAIT,
&context->tmfd, &updateCxt->lockmode,
&updateCxt->updateIndexes);
diff --git a/src/backend/libpq/pqmq.c b/src/backend/libpq/pqmq.c
index 22e5164adbf..1000b7bb06e 100644
--- a/src/backend/libpq/pqmq.c
+++ b/src/backend/libpq/pqmq.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/parallel.h"
+#include "commands/cluster.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqmq.h"
@@ -177,6 +178,10 @@ mq_putmessage(char msgtype, const char *s, size_t len)
SendProcSignal(pq_mq_parallel_leader_pid,
PROCSIG_PARALLEL_APPLY_MESSAGE,
pq_mq_parallel_leader_proc_number);
+ else if (IsRepackWorker())
+ SendProcSignal(pq_mq_parallel_leader_pid,
+ PROCSIG_REPACK_MESSAGE,
+ pq_mq_parallel_leader_proc_number);
else
{
Assert(IsParallelWorker());
diff --git a/src/backend/meson.build b/src/backend/meson.build
index 4f5292d8f88..5e3cfe2d6f8 100644
--- a/src/backend/meson.build
+++ b/src/backend/meson.build
@@ -219,5 +219,6 @@ pg_test_mod_args = pg_mod_args + {
subdir('jit/llvm')
subdir('replication/libpqwalreceiver')
subdir('replication/pgoutput')
+subdir('replication/pgoutput_repack')
subdir('snowball')
subdir('utils/mb/conversion_procs')
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 4d8b46adb46..cb9c60dfe31 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -13,6 +13,7 @@
#include "postgres.h"
#include "access/parallel.h"
+#include "commands/cluster.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -136,6 +137,10 @@ static const struct
.fn_name = "ParallelWorkerMain",
.fn_addr = ParallelWorkerMain
},
+ {
+ .fn_name = "RepackWorkerMain",
+ .fn_addr = RepackWorkerMain
+ },
{
.fn_name = "SequenceSyncWorkerMain",
.fn_addr = SequenceSyncWorkerMain
@@ -144,7 +149,6 @@ static const struct
.fn_name = "TableSyncWorkerMain",
.fn_addr = TableSyncWorkerMain
},
-
};
/* Private functions. */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 3c027bcb2f7..f172c0b2b40 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -33,6 +33,7 @@
#include "access/xlogreader.h"
#include "access/xlogrecord.h"
#include "catalog/pg_control.h"
+#include "commands/cluster.h"
#include "replication/decode.h"
#include "replication/logical.h"
#include "replication/message.h"
@@ -420,7 +421,8 @@ heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
case XLOG_HEAP2_MULTI_INSERT:
if (SnapBuildProcessChange(builder, xid, buf->origptr) &&
- !ctx->fast_forward)
+ !ctx->fast_forward &&
+ !change_useless_for_repack(buf))
DecodeMultiInsert(ctx, buf);
break;
case XLOG_HEAP2_NEW_CID:
@@ -465,6 +467,15 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
TransactionId xid = XLogRecGetXid(buf->record);
SnapBuild *builder = ctx->snapshot_builder;
+ /*
+ * XXX Should we return here if change_useless_for_repack() returns true,
+ * instead of calling the function below? Unlike the fast-forward case, we
+ * shouldn't need the base snapshot for the containing transaction until
+ * we receive a change that belongs to the table being REPACKed. Thus it
+ * should be fine to skip SnapBuildProcessChange(), and therefore
+ * reorderbuffer.c can create the transaction later.
+ */
+
ReorderBufferProcessXid(ctx->reorder, xid, buf->origptr);
/*
@@ -482,7 +493,8 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
case XLOG_HEAP_INSERT:
if (SnapBuildProcessChange(builder, xid, buf->origptr) &&
- !ctx->fast_forward)
+ !ctx->fast_forward &&
+ !change_useless_for_repack(buf))
DecodeInsert(ctx, buf);
break;
@@ -494,19 +506,22 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_HEAP_HOT_UPDATE:
case XLOG_HEAP_UPDATE:
if (SnapBuildProcessChange(builder, xid, buf->origptr) &&
- !ctx->fast_forward)
+ !ctx->fast_forward &&
+ !change_useless_for_repack(buf))
DecodeUpdate(ctx, buf);
break;
case XLOG_HEAP_DELETE:
if (SnapBuildProcessChange(builder, xid, buf->origptr) &&
- !ctx->fast_forward)
+ !ctx->fast_forward &&
+ !change_useless_for_repack(buf))
DecodeDelete(ctx, buf);
break;
case XLOG_HEAP_TRUNCATE:
if (SnapBuildProcessChange(builder, xid, buf->origptr) &&
- !ctx->fast_forward)
+ !ctx->fast_forward &&
+ !change_useless_for_repack(buf))
DecodeTruncate(ctx, buf);
break;
@@ -522,7 +537,8 @@ heap_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_HEAP_CONFIRM:
if (SnapBuildProcessChange(builder, xid, buf->origptr) &&
- !ctx->fast_forward)
+ !ctx->fast_forward &&
+ !change_useless_for_repack(buf))
DecodeSpecConfirm(ctx, buf);
break;
@@ -1019,6 +1035,15 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
xlrec = (xl_heap_delete *) XLogRecGetData(r);
+ /*
+ * Ignore changes which are considered useless for logical decoding.
+ * Currently such changes are created by REPACK (CONCURRENTLY) when
+ * replays DELETE commands on the new table (which is not yet visible to
+ * other transactions).
+ */
+ if (xlrec->flags & XLH_DELETE_NO_LOGICAL)
+ return;
+
/* only interested in our database */
XLogRecGetBlockTag(r, 0, &target_locator, NULL, NULL);
if (target_locator.dbOid != ctx->slot->data.database)
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 603a2b94d05..7651b187418 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -194,7 +194,11 @@ StartupDecodingContext(List *output_plugin_options,
ctx->slot = slot;
- ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, ctx);
+ /*
+ * TODO A separate patch for PG core, unless there's really a reason to
+ * pass ctx for private_data (May extensions expect ctx?).
+ */
+ ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, NULL);
if (!ctx->reader)
ereport(ERROR,
(errcode(ERRCODE_OUT_OF_MEMORY),
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 37f0c6028bd..9cf499ce7c6 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -440,7 +440,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder)
* for loading in different transaction.
*/
Snapshot
-SnapBuildInitialSnapshot(SnapBuild *builder)
+SnapBuildInitialSnapshot(SnapBuild *builder, bool repack)
{
Snapshot snap;
TransactionId xid;
@@ -448,7 +448,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
TransactionId *newxip;
int newxcnt = 0;
- Assert(XactIsoLevel == XACT_REPEATABLE_READ);
+ Assert(XactIsoLevel == XACT_REPEATABLE_READ || repack);
Assert(builder->building_full_snapshot);
/* don't allow older snapshots */
@@ -526,6 +526,11 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
snap->xcnt = newxcnt;
snap->xip = newxip;
+ /*
+ * FreeSnapshot() is more appropriate for REPACK than counting references.
+ */
+ snap->copied = repack;
+
return snap;
}
@@ -558,7 +563,7 @@ SnapBuildExportSnapshot(SnapBuild *builder)
XactIsoLevel = XACT_REPEATABLE_READ;
XactReadOnly = true;
- snap = SnapBuildInitialSnapshot(builder);
+ snap = SnapBuildInitialSnapshot(builder, false);
/*
* now that we've built a plain snapshot, make it active and use the
diff --git a/src/backend/replication/pgoutput_repack/Makefile b/src/backend/replication/pgoutput_repack/Makefile
new file mode 100644
index 00000000000..4efeb713b70
--- /dev/null
+++ b/src/backend/replication/pgoutput_repack/Makefile
@@ -0,0 +1,32 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for src/backend/replication/pgoutput_repack
+#
+# IDENTIFICATION
+# src/backend/replication/pgoutput_repack
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/replication/pgoutput_repack
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ $(WIN32RES) \
+ pgoutput_repack.o
+PGFILEDESC = "pgoutput_repack - logical replication output plugin for REPACK command"
+NAME = pgoutput_repack
+
+all: all-shared-lib
+
+include $(top_srcdir)/src/Makefile.shlib
+
+install: all installdirs install-lib
+
+installdirs: installdirs-lib
+
+uninstall: uninstall-lib
+
+clean distclean: clean-lib
+ rm -f $(OBJS)
diff --git a/src/backend/replication/pgoutput_repack/meson.build b/src/backend/replication/pgoutput_repack/meson.build
new file mode 100644
index 00000000000..6a88c0fb08d
--- /dev/null
+++ b/src/backend/replication/pgoutput_repack/meson.build
@@ -0,0 +1,18 @@
+# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+
+pgoutput_repack_sources = files(
+ 'pgoutput_repack.c',
+)
+
+if host_system == 'windows'
+ pgoutput_repack_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pgoutput_repack',
+ '--FILEDESC', 'pgoutput_repack - logical replication output plugin for REPACK command',])
+endif
+
+pgoutput_repack = shared_module('pgoutput_repack',
+ pgoutput_repack_sources,
+ kwargs: pg_mod_args,
+)
+
+backend_targets += pgoutput_repack
diff --git a/src/backend/replication/pgoutput_repack/pgoutput_repack.c b/src/backend/replication/pgoutput_repack/pgoutput_repack.c
new file mode 100644
index 00000000000..221260ca0a8
--- /dev/null
+++ b/src/backend/replication/pgoutput_repack/pgoutput_repack.c
@@ -0,0 +1,290 @@
+/*-------------------------------------------------------------------------
+ *
+ * pgoutput_repack.c
+ * Logical Replication output plugin for REPACK command
+ *
+ * Copyright (c) 2012-2026, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/pgoutput_repack/pgoutput_repack.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/detoast.h"
+#include "commands/repack_internal.h"
+#include "replication/snapbuild.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+static void repack_startup(LogicalDecodingContext *ctx,
+ OutputPluginOptions *opt, bool is_init);
+static void repack_shutdown(LogicalDecodingContext *ctx);
+static void repack_begin_txn(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void repack_commit_txn(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
+static void repack_process_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ Relation rel, ReorderBufferChange *change);
+static void repack_store_change(LogicalDecodingContext *ctx, Relation relation,
+ ConcurrentChangeKind kind, HeapTuple tuple);
+
+void
+_PG_output_plugin_init(OutputPluginCallbacks *cb)
+{
+ cb->startup_cb = repack_startup;
+ cb->begin_cb = repack_begin_txn;
+ cb->change_cb = repack_process_change;
+ cb->commit_cb = repack_commit_txn;
+ cb->shutdown_cb = repack_shutdown;
+}
+
+
+/* initialize this plugin */
+static void
+repack_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
+ bool is_init)
+{
+ ctx->output_plugin_private = NULL;
+
+ /* Probably unnecessary, as we don't use the SQL interface ... */
+ opt->output_type = OUTPUT_PLUGIN_BINARY_OUTPUT;
+
+ if (ctx->output_plugin_options != NIL)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("this plugin does not expect any options"));
+ }
+}
+
+static void
+repack_shutdown(LogicalDecodingContext *ctx)
+{
+}
+
+/*
+ * As we don't release the slot during processing of particular table, there's
+ * no room for SQL interface, even for debugging purposes. Therefore we need
+ * neither OutputPluginPrepareWrite() nor OutputPluginWrite() in the plugin
+ * callbacks. (Although we might want to write custom callbacks, this API
+ * seems to be unnecessarily generic for our purposes.)
+ */
+
+/* BEGIN callback */
+static void
+repack_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
+}
+
+/* COMMIT callback */
+static void
+repack_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+}
+
+/*
+ * Callback for individual changed tuples
+ */
+static void
+repack_process_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ RepackDecodingState *private PG_USED_FOR_ASSERTS_ONLY =
+ (RepackDecodingState *) ctx->output_writer_private;
+
+ /* Changes of other relation should not have been decoded. */
+ Assert(RelationGetRelid(relation) == private->relid);
+
+ /* Decode entry depending on its type */
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INSERT:
+ {
+ HeapTuple newtuple;
+
+ newtuple = change->data.tp.newtuple;
+
+ /*
+ * Identity checks in the main function should have made this
+ * impossible.
+ */
+ if (newtuple == NULL)
+ elog(ERROR, "incomplete insert info");
+
+ repack_store_change(ctx, relation, CHANGE_INSERT, newtuple);
+ }
+ break;
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ {
+ HeapTuple oldtuple,
+ newtuple;
+
+ oldtuple = change->data.tp.oldtuple;
+ newtuple = change->data.tp.newtuple;
+
+ if (newtuple == NULL)
+ elog(ERROR, "incomplete update info");
+
+ if (oldtuple != NULL)
+ repack_store_change(ctx, relation, CHANGE_UPDATE_OLD, oldtuple);
+
+ repack_store_change(ctx, relation, CHANGE_UPDATE_NEW, newtuple);
+ }
+ break;
+ case REORDER_BUFFER_CHANGE_DELETE:
+ {
+ HeapTuple oldtuple;
+
+ oldtuple = change->data.tp.oldtuple;
+
+ if (oldtuple == NULL)
+ elog(ERROR, "incomplete delete info");
+
+ repack_store_change(ctx, relation, CHANGE_DELETE, oldtuple);
+ }
+ break;
+ default:
+
+ /*
+ * Should not come here. This includes TRUNCATE of the table being
+ * processed. heap_decode() cannot check the file locator easily,
+ * but we assume that TRUNCATE uses AccessExclusiveLock on the
+ * table so it should not occur during REPACK (CONCURRENTLY).
+ */
+ Assert(false);
+ break;
+ }
+}
+
+/*
+ * Write the given tuple, with the given change kind, to the repack spill
+ * file. Later, the repack decoding worker can read these and replay
+ * the operations on the new copy of the table.
+ *
+ * For each change affecting the table being repacked, we store enough
+ * information about each tuple in it, so that it can be replayed in the
+ * new copy of the table.
+ *
+ * XXX for DELETE and the UPDATE OLD tuples, we could store just the
+ * replication identity instead of the full tuple.
+ */
+static void
+repack_store_change(LogicalDecodingContext *ctx, Relation relation,
+ ConcurrentChangeKind kind, HeapTuple tuple)
+{
+ RepackDecodingState *dstate;
+ MemoryContext oldcxt;
+ BufFile *file;
+ List *attrs_ext = NIL;
+ int natt_ext;
+
+ dstate = (RepackDecodingState *) ctx->output_writer_private;
+ file = dstate->file;
+
+ /* Store the change kind. */
+ BufFileWrite(file, &kind, 1);
+
+ /* Use a frequently-reset context to avoid dealing with leaks manually */
+ oldcxt = MemoryContextSwitchTo(dstate->change_cxt);
+
+ /*
+ * If the tuple contains "external indirect" attributes, we need to write
+ * the contents to the file because we have no control over that memory.
+ */
+ if (HeapTupleHasExternal(tuple))
+ {
+ TupleDesc desc = RelationGetDescr(relation);
+ TupleTableSlot *slot;
+
+ /* Initialize the slot, if not done already */
+ if (dstate->slot == NULL)
+ {
+ ResourceOwner saveResourceOwner;
+
+ MemoryContextSwitchTo(dstate->worker_cxt);
+ saveResourceOwner = CurrentResourceOwner;
+ CurrentResourceOwner = dstate->worker_resowner;
+ dstate->slot = MakeSingleTupleTableSlot(desc, &TTSOpsHeapTuple);
+ MemoryContextSwitchTo(dstate->change_cxt);
+ CurrentResourceOwner = saveResourceOwner;
+ }
+
+ slot = dstate->slot;
+ ExecStoreHeapTuple(tuple, slot, false);
+
+ /*
+ * Loop over all attributes, and find out which ones we need to spill
+ * separately, to wit: each one that's a non-null varlena and stored
+ * out of line.
+ */
+ for (int i = 0; i < desc->natts; i++)
+ {
+ CompactAttribute *attr = TupleDescCompactAttr(desc, i);
+ varlena *varlen;
+
+ if (attr->attisdropped || attr->attlen != -1 ||
+ slot_attisnull(slot, i + 1))
+ continue;
+
+ slot_getsomeattrs(slot, i + 1);
+
+ /*
+ * This is a non-null varlena datum, but we only care if it's
+ * out-of-line
+ */
+ varlen = (varlena *) DatumGetPointer(slot->tts_values[i]);
+ if (!VARATT_IS_EXTERNAL(varlen))
+ continue;
+
+ /*
+ * We spill any indirect-external attributes separately from the
+ * heap tuple. Anything else is written as is.
+ */
+ if (VARATT_IS_EXTERNAL_INDIRECT(varlen))
+ attrs_ext = lappend(attrs_ext, varlen);
+ else
+ {
+ /*
+ * Logical decoding should not produce "external expanded"
+ * attributes (those actually should never appear on disk), so
+ * only TOASTed attribute can be seen here.
+ *
+ * We get here if the table has external values but only
+ * in-line values are being updated now.
+ */
+ Assert(VARATT_IS_EXTERNAL_ONDISK(varlen));
+ }
+ }
+
+ ExecClearTuple(slot);
+ }
+
+ /*
+ * First, write the original heap tuple, prefixed by its length. Note
+ * that the external-toast tag for each toasted attribute will be present
+ * in what we write, so that we know where to restore each one later.
+ */
+ BufFileWrite(file, &tuple->t_len, sizeof(tuple->t_len));
+ BufFileWrite(file, tuple->t_data, tuple->t_len);
+
+ /* Then, write the number of external attributes we found. */
+ natt_ext = list_length(attrs_ext);
+ BufFileWrite(file, &natt_ext, sizeof(natt_ext));
+
+ /* Finally, the attributes themselves, if any */
+ foreach_ptr(varlena, attr_val, attrs_ext)
+ {
+ attr_val = detoast_external_attr(attr_val);
+ BufFileWrite(file, attr_val, VARSIZE_ANY(attr_val));
+ /* These attributes could be large, so free them right away */
+ pfree(attr_val);
+ }
+
+ /* Cleanup. */
+ MemoryContextSwitchTo(oldcxt);
+ MemoryContextReset(dstate->change_cxt);
+}
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 66507e9c2dd..fbbe09135bf 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1342,7 +1342,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
{
Snapshot snap;
- snap = SnapBuildInitialSnapshot(ctx->snapshot_builder);
+ snap = SnapBuildInitialSnapshot(ctx->snapshot_builder, false);
RestoreTransactionSnapshot(snap, MyProc);
}
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 7e017c8d53b..dd980145ced 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -19,6 +19,7 @@
#include "access/parallel.h"
#include "commands/async.h"
+#include "commands/cluster.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "port/pg_bitutils.h"
@@ -700,6 +701,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
+ if (CheckProcSignal(PROCSIG_REPACK_MESSAGE))
+ HandleRepackMessageInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT))
HandleRecoveryConflictInterrupt();
diff --git a/src/backend/storage/lmgr/generate-lwlocknames.pl b/src/backend/storage/lmgr/generate-lwlocknames.pl
index b49007167b0..2e7f1054e62 100644
--- a/src/backend/storage/lmgr/generate-lwlocknames.pl
+++ b/src/backend/storage/lmgr/generate-lwlocknames.pl
@@ -162,7 +162,7 @@ while (<$lwlocklist>)
die
"$wait_event_lwlocks[$lwlock_count] defined in wait_event_names.txt but "
- . " missing from lwlocklist.h"
+ . "missing from lwlocklist.h"
if $lwlock_count < scalar @wait_event_lwlocks;
die
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index b3563113219..4d253eddfa0 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -36,6 +36,7 @@
#include "access/xact.h"
#include "catalog/pg_type.h"
#include "commands/async.h"
+#include "commands/cluster.h"
#include "commands/event_trigger.h"
#include "commands/explain_state.h"
#include "commands/prepare.h"
@@ -3576,6 +3577,9 @@ ProcessInterrupts(void)
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
+
+ if (RepackMessagePending)
+ ProcessRepackMessages();
}
/*
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 6be80d2daad..e2f21349997 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -154,6 +154,7 @@ RECOVERY_CONFLICT_SNAPSHOT "Waiting for recovery conflict resolution for a vacuu
RECOVERY_CONFLICT_TABLESPACE "Waiting for recovery conflict resolution for dropping a tablespace."
RECOVERY_END_COMMAND "Waiting for <xref linkend="guc-recovery-end-command"/> to complete."
RECOVERY_PAUSE "Waiting for recovery to be resumed."
+REPACK_WORKER_EXPORT "Waiting for decoding worker to export a new output file."
REPLICATION_ORIGIN_DROP "Waiting for a replication origin to become inactive so it can be dropped."
REPLICATION_SLOT_DROP "Waiting for a replication slot to become inactive so it can be dropped."
RESTORE_COMMAND "Waiting for <xref linkend="guc-restore-command"/> to complete."
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 2e6197f5f35..e0129df3e40 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -215,7 +215,6 @@ static List *exportedSnapshots = NIL;
/* Prototypes for local functions */
static Snapshot CopySnapshot(Snapshot snapshot);
static void UnregisterSnapshotNoOwner(Snapshot snapshot);
-static void FreeSnapshot(Snapshot snapshot);
static void SnapshotResetXmin(void);
/* ResourceOwner callbacks to track snapshot references */
@@ -660,7 +659,7 @@ CopySnapshot(Snapshot snapshot)
* FreeSnapshot
* Free the memory associated with a snapshot.
*/
-static void
+void
FreeSnapshot(Snapshot snapshot)
{
Assert(snapshot->regd_count == 0);
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index 523d3f39fc5..e41c965a1cc 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -5227,8 +5227,8 @@ match_previous_words(int pattern_id,
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("ANALYZE", "VERBOSE");
- else if (TailMatches("ANALYZE", "VERBOSE"))
+ COMPLETE_WITH("ANALYZE", "CONCURRENTLY", "VERBOSE");
+ else if (TailMatches("ANALYZE", "CONCURRENTLY", "VERBOSE"))
COMPLETE_WITH("ON", "OFF");
}
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 9b403203006..67d113de10b 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -384,13 +384,13 @@ extern void heap_multi_insert(Relation relation, TupleTableSlot **slots,
int ntuples, CommandId cid, int options,
BulkInsertState bistate);
extern TM_Result heap_delete(Relation relation, const ItemPointerData *tid,
- CommandId cid, Snapshot crosscheck, bool wait,
- TM_FailureData *tmfd, bool changingPart);
+ CommandId cid, Snapshot crosscheck, int options,
+ TM_FailureData *tmfd);
extern void heap_finish_speculative(Relation relation, const ItemPointerData *tid);
extern void heap_abort_speculative(Relation relation, const ItemPointerData *tid);
extern TM_Result heap_update(Relation relation, const ItemPointerData *otid,
HeapTuple newtup,
- CommandId cid, Snapshot crosscheck, bool wait,
+ CommandId cid, Snapshot crosscheck, int options,
TM_FailureData *tmfd, LockTupleMode *lockmode,
TU_UpdateIndexes *update_indexes);
extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 516806fcca2..fdca7d821c8 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -104,6 +104,8 @@
#define XLH_DELETE_CONTAINS_OLD_KEY (1<<2)
#define XLH_DELETE_IS_SUPER (1<<3)
#define XLH_DELETE_IS_PARTITION_MOVE (1<<4)
+/* See heap_delete() */
+#define XLH_DELETE_NO_LOGICAL (1<<5)
/* convenience macro for checking whether any form of old tuple was logged */
#define XLH_DELETE_CONTAINS_OLD \
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 06084752245..1e51e22344f 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -260,6 +260,15 @@ typedef struct TM_IndexDeleteOp
#define TABLE_INSERT_FROZEN 0x0004
#define TABLE_INSERT_NO_LOGICAL 0x0008
+/* "options" flag bits for table_tuple_update */
+#define TABLE_UPDATE_WAIT 0x0001
+#define TABLE_UPDATE_NO_LOGICAL 0x0002
+
+/* "options" flag bits for table_tuple_delete */
+#define TABLE_DELETE_WAIT 0x0001
+#define TABLE_DELETE_CHANGING_PART 0x0002
+#define TABLE_DELETE_NO_LOGICAL 0x0004
+
/* flag bits for table_tuple_lock */
/* Follow tuples whose update is in progress if lock modes don't conflict */
#define TUPLE_LOCK_FLAG_LOCK_UPDATE_IN_PROGRESS (1 << 0)
@@ -535,9 +544,8 @@ typedef struct TableAmRoutine
CommandId cid,
Snapshot snapshot,
Snapshot crosscheck,
- bool wait,
- TM_FailureData *tmfd,
- bool changingPart);
+ int options,
+ TM_FailureData *tmfd);
/* see table_tuple_update() for reference about parameters */
TM_Result (*tuple_update) (Relation rel,
@@ -546,7 +554,7 @@ typedef struct TableAmRoutine
CommandId cid,
Snapshot snapshot,
Snapshot crosscheck,
- bool wait,
+ int options,
TM_FailureData *tmfd,
LockTupleMode *lockmode,
TU_UpdateIndexes *update_indexes);
@@ -629,6 +637,7 @@ typedef struct TableAmRoutine
Relation OldIndex,
bool use_sort,
TransactionId OldestXmin,
+ Snapshot snapshot,
TransactionId *xid_cutoff,
MultiXactId *multi_cutoff,
double *num_tuples,
@@ -1459,6 +1468,7 @@ table_multi_insert(Relation rel, TupleTableSlot **slots, int nslots,
* cid - delete command ID (used for visibility test, and stored into
* cmax if successful)
* crosscheck - if not InvalidSnapshot, also check tuple against this
+ * XXX document options
* wait - true if should wait for any conflicting update to commit/abort
* changingPart - true iff the tuple is being moved to another partition
* table due to an update of the partition key. Otherwise, false.
@@ -1476,12 +1486,12 @@ table_multi_insert(Relation rel, TupleTableSlot **slots, int nslots,
*/
static inline TM_Result
table_tuple_delete(Relation rel, ItemPointer tid, CommandId cid,
- Snapshot snapshot, Snapshot crosscheck, bool wait,
- TM_FailureData *tmfd, bool changingPart)
+ Snapshot snapshot, Snapshot crosscheck, int options,
+ TM_FailureData *tmfd)
{
return rel->rd_tableam->tuple_delete(rel, tid, cid,
snapshot, crosscheck,
- wait, tmfd, changingPart);
+ options, tmfd);
}
/*
@@ -1496,7 +1506,12 @@ table_tuple_delete(Relation rel, ItemPointer tid, CommandId cid,
* cid - update command ID (used for visibility test, and stored into
* cmax/cmin if successful)
* crosscheck - if not InvalidSnapshot, also check old tuple against this
- * wait - true if should wait for any conflicting update to commit/abort
+ * options - These allow the caller to specify options that may change the
+ * behavior of the AM. The AM will ignore options that it does not support.
+ * TABLE_UPDATE_WAIT -- set if should wait for any conflicting update to
+ * commit/abort
+ * TABLE_UPDATE_NO_LOGICAL -- force-disables the emitting of logical
+ * decoding information for the tuple.
*
* Output parameters:
* slot - newly constructed tuple data to store
@@ -1522,12 +1537,12 @@ table_tuple_delete(Relation rel, ItemPointer tid, CommandId cid,
static inline TM_Result
table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot,
CommandId cid, Snapshot snapshot, Snapshot crosscheck,
- bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode,
+ int options, TM_FailureData *tmfd, LockTupleMode *lockmode,
TU_UpdateIndexes *update_indexes)
{
return rel->rd_tableam->tuple_update(rel, otid, slot,
cid, snapshot, crosscheck,
- wait, tmfd,
+ options, tmfd,
lockmode, update_indexes);
}
@@ -1657,6 +1672,8 @@ table_relation_copy_data(Relation rel, const RelFileLocator *newrlocator)
* not needed for the relation's AM
* - *xid_cutoff - ditto
* - *multi_cutoff - ditto
+ * - snapshot - if != NULL, ignore data changes done by transactions that this
+ * (MVCC) snapshot considers still in-progress or in the future.
*
* Output parameters:
* - *xid_cutoff - rel's new relfrozenxid value, may be invalid
@@ -1669,6 +1686,7 @@ table_relation_copy_for_cluster(Relation OldTable, Relation NewTable,
Relation OldIndex,
bool use_sort,
TransactionId OldestXmin,
+ Snapshot snapshot,
TransactionId *xid_cutoff,
MultiXactId *multi_cutoff,
double *num_tuples,
@@ -1677,6 +1695,7 @@ table_relation_copy_for_cluster(Relation OldTable, Relation NewTable,
{
OldTable->rd_tableam->relation_copy_for_cluster(OldTable, NewTable, OldIndex,
use_sort, OldestXmin,
+ snapshot,
xid_cutoff, multi_cutoff,
num_tuples, tups_vacuumed,
tups_recently_dead);
diff --git a/src/include/commands/cluster.h b/src/include/commands/cluster.h
index c79cd2d0e19..8ed7816e7b4 100644
--- a/src/include/commands/cluster.h
+++ b/src/include/commands/cluster.h
@@ -13,6 +13,8 @@
#ifndef CLUSTER_H
#define CLUSTER_H
+#include <signal.h>
+
#include "nodes/parsenodes.h"
#include "parser/parse_node.h"
#include "storage/lockdefs.h"
@@ -25,6 +27,7 @@
#define CLUOPT_RECHECK_ISCLUSTERED 0x04 /* recheck relation state for
* indisclustered */
#define CLUOPT_ANALYZE 0x08 /* do an ANALYZE */
+#define CLUOPT_CONCURRENT 0x10 /* allow concurrent data changes */
/* options for CLUSTER */
typedef struct ClusterParams
@@ -32,11 +35,13 @@ typedef struct ClusterParams
bits32 options; /* bitmask of CLUOPT_* */
} ClusterParams;
+extern PGDLLIMPORT volatile sig_atomic_t RepackMessagePending;
+
extern void ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel);
extern void cluster_rel(RepackCommand command, Relation OldHeap, Oid indexOid,
- ClusterParams *params);
+ ClusterParams *params, bool isTopLevel);
extern void check_index_is_clusterable(Relation OldHeap, Oid indexOid,
LOCKMODE lockmode);
extern void mark_index_clustered(Relation rel, Oid indexOid, bool is_internal);
@@ -48,8 +53,17 @@ extern void finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
bool swap_toast_by_content,
bool check_constraints,
bool is_internal,
+ bool reindex,
TransactionId frozenXid,
MultiXactId cutoffMulti,
char newrelpersistence);
+extern void HandleRepackMessageInterrupt(void);
+extern void ProcessRepackMessages(void);
+
+/* in repack_worker.c */
+extern void RepackWorkerMain(Datum main_arg);
+extern bool IsRepackWorker(void);
+
+
#endif /* CLUSTER_H */
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index 9c40772706c..b5b01c1bb6d 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -86,10 +86,12 @@
#define PROGRESS_REPACK_PHASE 1
#define PROGRESS_REPACK_INDEX_RELID 2
#define PROGRESS_REPACK_HEAP_TUPLES_SCANNED 3
-#define PROGRESS_REPACK_HEAP_TUPLES_WRITTEN 4
-#define PROGRESS_REPACK_TOTAL_HEAP_BLKS 5
-#define PROGRESS_REPACK_HEAP_BLKS_SCANNED 6
-#define PROGRESS_REPACK_INDEX_REBUILD_COUNT 7
+#define PROGRESS_REPACK_HEAP_TUPLES_INSERTED 4
+#define PROGRESS_REPACK_HEAP_TUPLES_UPDATED 5
+#define PROGRESS_REPACK_HEAP_TUPLES_DELETED 6
+#define PROGRESS_REPACK_TOTAL_HEAP_BLKS 7
+#define PROGRESS_REPACK_HEAP_BLKS_SCANNED 8
+#define PROGRESS_REPACK_INDEX_REBUILD_COUNT 9
/*
* Phases of repack (as advertised via PROGRESS_REPACK_PHASE).
@@ -98,9 +100,10 @@
#define PROGRESS_REPACK_PHASE_INDEX_SCAN_HEAP 2
#define PROGRESS_REPACK_PHASE_SORT_TUPLES 3
#define PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP 4
-#define PROGRESS_REPACK_PHASE_SWAP_REL_FILES 5
-#define PROGRESS_REPACK_PHASE_REBUILD_INDEX 6
-#define PROGRESS_REPACK_PHASE_FINAL_CLEANUP 7
+#define PROGRESS_REPACK_PHASE_CATCH_UP 5
+#define PROGRESS_REPACK_PHASE_SWAP_REL_FILES 6
+#define PROGRESS_REPACK_PHASE_REBUILD_INDEX 7
+#define PROGRESS_REPACK_PHASE_FINAL_CLEANUP 8
/* Progress parameters for CREATE INDEX */
/* 3, 4 and 5 reserved for "waitfor" metrics */
diff --git a/src/include/commands/repack_internal.h b/src/include/commands/repack_internal.h
new file mode 100644
index 00000000000..d3402e0dedd
--- /dev/null
+++ b/src/include/commands/repack_internal.h
@@ -0,0 +1,128 @@
+/*-------------------------------------------------------------------------
+ *
+ * repack_internal.h
+ * header for REPACK internals
+ *
+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994-5, Regents of the University of California
+ *
+ * src/include/commands/repack_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef REPACK_INTERNAL_H
+#define REPACK_INTERNAL_H
+
+#include "nodes/execnodes.h"
+#include "replication/decode.h"
+#include "postmaster/bgworker.h"
+#include "replication/logical.h"
+#include "storage/buffile.h"
+#include "storage/sharedfileset.h"
+#include "storage/shm_mq.h"
+#include "utils/resowner.h"
+
+/*
+ * Stored as a single byte in the output file.
+ */
+typedef char ConcurrentChangeKind;
+
+#define CHANGE_INSERT 'i'
+#define CHANGE_UPDATE_OLD 'u'
+#define CHANGE_UPDATE_NEW 'U'
+#define CHANGE_DELETE 'd'
+
+extern pid_t backend_pid;
+extern ProcNumber backend_proc_number;
+
+/*
+ * Logical decoding state.
+ *
+ * The output plugin uses it to store the data changes that it decodes from
+ * WAL while the table contents is being copied to a new storage.
+ */
+typedef struct RepackDecodingState
+{
+#ifdef USE_ASSERT_CHECKING
+ /* The relation whose changes we're decoding. */
+ Oid relid;
+#endif
+
+ /* Per-change memory context. */
+ MemoryContext change_cxt;
+
+ /* A tuple slot used to pass tuples back and forth */
+ TupleTableSlot *slot;
+
+ /*
+ * Memory context and resource owner of the decoding worker's transaction.
+ */
+ MemoryContext worker_cxt;
+ ResourceOwner worker_resowner;
+
+ /* The current output file. */
+ BufFile *file;
+} RepackDecodingState;
+
+/*
+ * Layout of shared memory used for communication between backend and the
+ * worker that performs logical decoding of data changes
+ */
+typedef struct DecodingWorkerShared
+{
+ /* Is the decoding initialized? */
+ bool initialized;
+
+ /*
+ * Once the worker has reached this LSN, it should close the current
+ * output file and either create a new one or exit, according to the field
+ * 'done'. If the value is InvalidXLogRecPtr, the worker should decode all
+ * the WAL available and keep checking this field. It is ok if the worker
+ * had already decoded records whose LSN is >= lsn_upto before this field
+ * has been set.
+ */
+ XLogRecPtr lsn_upto;
+
+ /* Exit after closing the current file? */
+ bool done;
+
+ /* The output is stored here. */
+ SharedFileSet sfs;
+
+ /* Number of the last file exported by the worker. */
+ int last_exported;
+
+ /* Synchronize access to the fields above. */
+ slock_t mutex;
+
+ /* Database to connect to. */
+ Oid dbid;
+
+ /* Role to connect as. */
+ Oid roleid;
+
+ /* Decode data changes of this relation. */
+ Oid relid;
+
+ /* The backend uses this to wait for the worker. */
+ ConditionVariable cv;
+
+ /* Info to signal the backend. */
+ PGPROC *backend_proc;
+ pid_t backend_pid;
+ ProcNumber backend_proc_number;
+
+ /*
+ * Memory the queue is located in.
+ *
+ * For considerations on the value see the comments of
+ * PARALLEL_ERROR_QUEUE_SIZE.
+ */
+#define REPACK_ERROR_QUEUE_SIZE 16384
+ char error_queue[FLEXIBLE_ARRAY_MEMBER];
+} DecodingWorkerShared;
+
+extern void DecodingWorkerFileName(char *fname, Oid relid, uint32 seq);
+
+
+#endif /* REPACK_INTERNAL_H */
diff --git a/src/include/replication/decode.h b/src/include/replication/decode.h
index 49f00fc48b8..02b5393474c 100644
--- a/src/include/replication/decode.h
+++ b/src/include/replication/decode.h
@@ -31,4 +31,8 @@ extern void logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf
extern void LogicalDecodingProcessRecord(LogicalDecodingContext *ctx,
XLogReaderState *record);
+/* in commands/cluster.c */
+extern bool change_useless_for_repack(XLogRecordBuffer *buf);
+
+
#endif
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index ccded021433..2b84f0058f0 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -72,7 +72,7 @@ extern void FreeSnapshotBuilder(SnapBuild *builder);
extern void SnapBuildSnapDecRefcount(Snapshot snap);
-extern Snapshot SnapBuildInitialSnapshot(SnapBuild *builder);
+extern Snapshot SnapBuildInitialSnapshot(SnapBuild *builder, bool repack);
extern const char *SnapBuildExportSnapshot(SnapBuild *builder);
extern void SnapBuildClearExportedSnapshot(void);
extern void SnapBuildResetExportedSnapshotState(void);
diff --git a/src/include/storage/lockdefs.h b/src/include/storage/lockdefs.h
index b73bb5618e6..3785b009808 100644
--- a/src/include/storage/lockdefs.h
+++ b/src/include/storage/lockdefs.h
@@ -36,8 +36,8 @@ typedef int LOCKMODE;
#define AccessShareLock 1 /* SELECT */
#define RowShareLock 2 /* SELECT FOR UPDATE/FOR SHARE */
#define RowExclusiveLock 3 /* INSERT, UPDATE, DELETE */
-#define ShareUpdateExclusiveLock 4 /* VACUUM (non-FULL), ANALYZE, CREATE
- * INDEX CONCURRENTLY */
+#define ShareUpdateExclusiveLock 4 /* VACUUM (non-exclusive), ANALYZE, CREATE
+ * INDEX CONCURRENTLY, REPACK CONCURRENTLY */
#define ShareLock 5 /* CREATE INDEX (WITHOUT CONCURRENTLY) */
#define ShareRowExclusiveLock 6 /* like EXCLUSIVE MODE, but allows ROW
* SHARE */
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a93..a944ee0d211 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -36,6 +36,7 @@ typedef enum
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
+ PROCSIG_REPACK_MESSAGE, /* Message from repack worker */
PROCSIG_RECOVERY_CONFLICT, /* backend is blocking recovery, check
* PGPROC->pendingRecoveryConflicts for the
* reason */
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 1c550096393..55ecc2c2054 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -78,6 +78,8 @@ extern Snapshot GetTransactionSnapshot(void);
extern Snapshot GetLatestSnapshot(void);
extern void SnapshotSetCommandId(CommandId curcid);
+extern void FreeSnapshot(Snapshot snapshot);
+
extern Snapshot GetCatalogSnapshot(Oid relid);
extern Snapshot GetNonHistoricCatalogSnapshot(Oid relid);
extern void InvalidateCatalogSnapshot(void);
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index a41d781f8c9..2cd7d87c533 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -14,6 +14,8 @@ REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
ISOLATION = basic \
inplace \
+ repack \
+ repack_toast \
syscache-update-pruned \
heap_lock_update
diff --git a/src/test/modules/injection_points/expected/repack.out b/src/test/modules/injection_points/expected/repack.out
new file mode 100644
index 00000000000..b575e9052ee
--- /dev/null
+++ b/src/test/modules/injection_points/expected/repack.out
@@ -0,0 +1,113 @@
+Parsed test spec with 2 sessions
+
+starting permutation: wait_before_lock change_existing change_new change_subxact1 change_subxact2 check2 wakeup_before_lock check1
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step change_existing:
+ UPDATE repack_test SET i=10 where i=1;
+ UPDATE repack_test SET j=20 where i=2;
+ UPDATE repack_test SET i=30 where i=3;
+ UPDATE repack_test SET i=40 where i=30;
+ DELETE FROM repack_test WHERE i=4;
+
+step change_new:
+ INSERT INTO repack_test(i, j) VALUES (5, 5), (6, 6), (7, 7), (8, 8);
+ UPDATE repack_test SET i=50 where i=5;
+ UPDATE repack_test SET j=60 where i=6;
+ DELETE FROM repack_test WHERE i=7;
+
+step change_subxact1:
+ BEGIN;
+ INSERT INTO repack_test(i, j) VALUES (100, 100);
+ SAVEPOINT s1;
+ UPDATE repack_test SET i=101 where i=100;
+ SAVEPOINT s2;
+ UPDATE repack_test SET i=102 where i=101;
+ COMMIT;
+
+step change_subxact2:
+ BEGIN;
+ SAVEPOINT s1;
+ INSERT INTO repack_test(i, j) VALUES (110, 110);
+ ROLLBACK TO SAVEPOINT s1;
+ INSERT INTO repack_test(i, j) VALUES (110, 111);
+ COMMIT;
+
+step check2:
+ INSERT INTO relfilenodes(node)
+ SELECT relfilenode FROM pg_class WHERE relname='repack_test';
+
+ SELECT i, j FROM repack_test ORDER BY i, j;
+
+ INSERT INTO data_s2(i, j)
+ SELECT i, j FROM repack_test;
+
+ i| j
+---+---
+ 2| 20
+ 6| 60
+ 8| 8
+ 10| 1
+ 40| 3
+ 50| 5
+102|100
+110|111
+(8 rows)
+
+step wakeup_before_lock:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step check1:
+ INSERT INTO relfilenodes(node)
+ SELECT relfilenode FROM pg_class WHERE relname='repack_test';
+
+ SELECT count(DISTINCT node) FROM relfilenodes;
+
+ SELECT i, j FROM repack_test ORDER BY i, j;
+
+ INSERT INTO data_s1(i, j)
+ SELECT i, j FROM repack_test;
+
+ SELECT count(*)
+ FROM data_s1 d1 FULL JOIN data_s2 d2 USING (i, j)
+ WHERE d1.i ISNULL OR d2.i ISNULL;
+
+count
+-----
+ 2
+(1 row)
+
+ i| j
+---+---
+ 2| 20
+ 6| 60
+ 8| 8
+ 10| 1
+ 40| 3
+ 50| 5
+102|100
+110|111
+(8 rows)
+
+count
+-----
+ 0
+(1 row)
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
diff --git a/src/test/modules/injection_points/expected/repack_toast.out b/src/test/modules/injection_points/expected/repack_toast.out
new file mode 100644
index 00000000000..b56dde134f8
--- /dev/null
+++ b/src/test/modules/injection_points/expected/repack_toast.out
@@ -0,0 +1,65 @@
+Parsed test spec with 2 sessions
+
+starting permutation: wait_before_lock change check2 wakeup_before_lock check1
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test;
+ <waiting ...>
+step change:
+ UPDATE repack_test SET j=get_long_string() where i=2;
+ DELETE FROM repack_test WHERE i=3;
+ INSERT INTO repack_test(i, j) VALUES (4, get_long_string());
+ UPDATE repack_test SET i=3 where i=1;
+
+step check2:
+ INSERT INTO relfilenodes(node)
+ SELECT c2.relfilenode
+ FROM pg_class c1 JOIN pg_class c2 ON c2.oid = c1.oid OR c2.oid = c1.reltoastrelid
+ WHERE c1.relname='repack_test';
+
+ INSERT INTO data_s2(i, j)
+ SELECT i, j FROM repack_test;
+
+step wakeup_before_lock:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step check1:
+ INSERT INTO relfilenodes(node)
+ SELECT c2.relfilenode
+ FROM pg_class c1 JOIN pg_class c2 ON c2.oid = c1.oid OR c2.oid = c1.reltoastrelid
+ WHERE c1.relname='repack_test';
+
+ SELECT count(DISTINCT node) FROM relfilenodes;
+
+ INSERT INTO data_s1(i, j)
+ SELECT i, j FROM repack_test;
+
+ SELECT count(*)
+ FROM data_s1 d1 FULL JOIN data_s2 d2 USING (i, j)
+ WHERE d1.i ISNULL OR d2.i ISNULL;
+
+count
+-----
+ 4
+(1 row)
+
+count
+-----
+ 0
+(1 row)
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index fcc85414515..a414abb924b 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -45,6 +45,8 @@ tests += {
'specs': [
'basic',
'inplace',
+ 'repack',
+ 'repack_toast',
'syscache-update-pruned',
'heap_lock_update',
],
diff --git a/src/test/modules/injection_points/specs/repack.spec b/src/test/modules/injection_points/specs/repack.spec
new file mode 100644
index 00000000000..d727a9b056b
--- /dev/null
+++ b/src/test/modules/injection_points/specs/repack.spec
@@ -0,0 +1,142 @@
+# REPACK (CONCURRENTLY) ... USING INDEX ...;
+setup
+{
+ CREATE EXTENSION injection_points;
+
+ CREATE TABLE repack_test(i int PRIMARY KEY, j int);
+ INSERT INTO repack_test(i, j) VALUES (1, 1), (2, 2), (3, 3), (4, 4);
+
+ CREATE TABLE relfilenodes(node oid);
+
+ CREATE TABLE data_s1(i int, j int);
+ CREATE TABLE data_s2(i int, j int);
+}
+
+teardown
+{
+ DROP TABLE repack_test;
+ DROP EXTENSION injection_points;
+
+ DROP TABLE relfilenodes;
+ DROP TABLE data_s1;
+ DROP TABLE data_s2;
+}
+
+session s1
+setup
+{
+ SELECT injection_points_set_local();
+ SELECT injection_points_attach('repack-concurrently-before-lock', 'wait');
+}
+# Perform the initial load and wait for s2 to do some data changes.
+step wait_before_lock
+{
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+}
+# Check the table from the perspective of s1.
+#
+# Besides the contents, we also check that relfilenode has changed.
+
+# Have each session write the contents into a table and use FULL JOIN to check
+# if the outputs are identical.
+step check1
+{
+ INSERT INTO relfilenodes(node)
+ SELECT relfilenode FROM pg_class WHERE relname='repack_test';
+
+ SELECT count(DISTINCT node) FROM relfilenodes;
+
+ SELECT i, j FROM repack_test ORDER BY i, j;
+
+ INSERT INTO data_s1(i, j)
+ SELECT i, j FROM repack_test;
+
+ SELECT count(*)
+ FROM data_s1 d1 FULL JOIN data_s2 d2 USING (i, j)
+ WHERE d1.i ISNULL OR d2.i ISNULL;
+}
+teardown
+{
+ SELECT injection_points_detach('repack-concurrently-before-lock');
+}
+
+session s2
+# Change the existing data. UPDATE changes both key and non-key columns. Also
+# update one row twice to test whether tuple version generated by this session
+# can be found.
+step change_existing
+{
+ UPDATE repack_test SET i=10 where i=1;
+ UPDATE repack_test SET j=20 where i=2;
+ UPDATE repack_test SET i=30 where i=3;
+ UPDATE repack_test SET i=40 where i=30;
+ DELETE FROM repack_test WHERE i=4;
+}
+# Insert new rows and UPDATE / DELETE some of them. Again, update both key and
+# non-key column.
+step change_new
+{
+ INSERT INTO repack_test(i, j) VALUES (5, 5), (6, 6), (7, 7), (8, 8);
+ UPDATE repack_test SET i=50 where i=5;
+ UPDATE repack_test SET j=60 where i=6;
+ DELETE FROM repack_test WHERE i=7;
+}
+
+# When applying concurrent data changes, we should see the effects of an
+# in-progress subtransaction.
+#
+# XXX Not sure this test is useful now - it was designed for the patch that
+# preserves tuple visibility and which therefore modifies
+# TransactionIdIsCurrentTransactionId().
+step change_subxact1
+{
+ BEGIN;
+ INSERT INTO repack_test(i, j) VALUES (100, 100);
+ SAVEPOINT s1;
+ UPDATE repack_test SET i=101 where i=100;
+ SAVEPOINT s2;
+ UPDATE repack_test SET i=102 where i=101;
+ COMMIT;
+}
+
+# When applying concurrent data changes, we should not see the effects of a
+# rolled back subtransaction.
+#
+# XXX Is this test useful? See above.
+step change_subxact2
+{
+ BEGIN;
+ SAVEPOINT s1;
+ INSERT INTO repack_test(i, j) VALUES (110, 110);
+ ROLLBACK TO SAVEPOINT s1;
+ INSERT INTO repack_test(i, j) VALUES (110, 111);
+ COMMIT;
+}
+
+# Check the table from the perspective of s2.
+step check2
+{
+ INSERT INTO relfilenodes(node)
+ SELECT relfilenode FROM pg_class WHERE relname='repack_test';
+
+ SELECT i, j FROM repack_test ORDER BY i, j;
+
+ INSERT INTO data_s2(i, j)
+ SELECT i, j FROM repack_test;
+}
+step wakeup_before_lock
+{
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+}
+
+# Test if data changes introduced while one session is performing REPACK
+# CONCURRENTLY find their way into the table.
+permutation
+ wait_before_lock
+ change_existing
+ change_new
+ change_subxact1
+ change_subxact2
+ check2
+ wakeup_before_lock
+ check1
diff --git a/src/test/modules/injection_points/specs/repack_toast.spec b/src/test/modules/injection_points/specs/repack_toast.spec
new file mode 100644
index 00000000000..b878b198971
--- /dev/null
+++ b/src/test/modules/injection_points/specs/repack_toast.spec
@@ -0,0 +1,112 @@
+# REPACK (CONCURRENTLY);
+#
+# Test handling of TOAST. At the same time, no tuplesort.
+setup
+{
+ CREATE EXTENSION injection_points;
+
+ -- Return a string that needs to be TOASTed.
+ CREATE FUNCTION get_long_string()
+ RETURNS text
+ LANGUAGE sql as $$
+ SELECT string_agg(chr(65 + trunc(25 * random())::int), '')
+ FROM generate_series(1, 2048) s(x);
+ $$;
+
+ CREATE TABLE repack_test(i int PRIMARY KEY, j text);
+ INSERT INTO repack_test(i, j) VALUES (1, get_long_string()),
+ (2, get_long_string()), (3, get_long_string());
+
+ CREATE TABLE relfilenodes(node oid);
+
+ CREATE TABLE data_s1(i int, j text);
+ CREATE TABLE data_s2(i int, j text);
+}
+
+teardown
+{
+ DROP TABLE repack_test;
+ DROP EXTENSION injection_points;
+ DROP FUNCTION get_long_string();
+
+ DROP TABLE relfilenodes;
+ DROP TABLE data_s1;
+ DROP TABLE data_s2;
+}
+
+session s1
+setup
+{
+ SELECT injection_points_set_local();
+ SELECT injection_points_attach('repack-concurrently-before-lock', 'wait');
+}
+# Perform the initial load and wait for s2 to do some data changes.
+step wait_before_lock
+{
+ REPACK (CONCURRENTLY) repack_test;
+}
+# Check the table from the perspective of s1.
+#
+# Besides the contents, we also check that relfilenode has changed.
+
+# Have each session write the contents into a table and use FULL JOIN to check
+# if the outputs are identical.
+step check1
+{
+ INSERT INTO relfilenodes(node)
+ SELECT c2.relfilenode
+ FROM pg_class c1 JOIN pg_class c2 ON c2.oid = c1.oid OR c2.oid = c1.reltoastrelid
+ WHERE c1.relname='repack_test';
+
+ SELECT count(DISTINCT node) FROM relfilenodes;
+
+ INSERT INTO data_s1(i, j)
+ SELECT i, j FROM repack_test;
+
+ SELECT count(*)
+ FROM data_s1 d1 FULL JOIN data_s2 d2 USING (i, j)
+ WHERE d1.i ISNULL OR d2.i ISNULL;
+}
+teardown
+{
+ SELECT injection_points_detach('repack-concurrently-before-lock');
+}
+
+session s2
+step change
+# Separately test UPDATE of both plain ("i") and TOASTed ("j") attribute. In
+# the first case, the new tuple we get from reorderbuffer.c contains "j" as a
+# TOAST pointer, which we need to update so it points to the new heap. In the
+# latter case, we receive "j" as "external indirect" value - here we test that
+# the decoding worker writes the tuple to a file correctly and that the
+# backend executing REPACK manages to restore it.
+{
+ UPDATE repack_test SET j=get_long_string() where i=2;
+ DELETE FROM repack_test WHERE i=3;
+ INSERT INTO repack_test(i, j) VALUES (4, get_long_string());
+ UPDATE repack_test SET i=3 where i=1;
+}
+# Check the table from the perspective of s2.
+step check2
+{
+ INSERT INTO relfilenodes(node)
+ SELECT c2.relfilenode
+ FROM pg_class c1 JOIN pg_class c2 ON c2.oid = c1.oid OR c2.oid = c1.reltoastrelid
+ WHERE c1.relname='repack_test';
+
+ INSERT INTO data_s2(i, j)
+ SELECT i, j FROM repack_test;
+}
+step wakeup_before_lock
+{
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+}
+
+# Test if data changes introduced while one session is performing REPACK
+# CONCURRENTLY find their way into the table.
+permutation
+ wait_before_lock
+ change
+ check2
+ wakeup_before_lock
+ check1
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2b3cf6d8569..84d61f3ab34 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2021,7 +2021,7 @@ pg_stat_progress_cluster| SELECT pid,
phase,
repack_index_relid AS cluster_index_relid,
heap_tuples_scanned,
- heap_tuples_written,
+ (heap_tuples_inserted + heap_tuples_updated) AS heap_tuples_written,
heap_blks_total,
heap_blks_scanned,
index_rebuild_count
@@ -2101,17 +2101,20 @@ pg_stat_progress_repack| SELECT s.pid,
WHEN 2 THEN 'index scanning heap'::text
WHEN 3 THEN 'sorting tuples'::text
WHEN 4 THEN 'writing new heap'::text
- WHEN 5 THEN 'swapping relation files'::text
- WHEN 6 THEN 'rebuilding index'::text
- WHEN 7 THEN 'performing final cleanup'::text
+ WHEN 5 THEN 'catch-up'::text
+ WHEN 6 THEN 'swapping relation files'::text
+ WHEN 7 THEN 'rebuilding index'::text
+ WHEN 8 THEN 'performing final cleanup'::text
ELSE NULL::text
END AS phase,
(s.param3)::oid AS repack_index_relid,
s.param4 AS heap_tuples_scanned,
- s.param5 AS heap_tuples_written,
- s.param6 AS heap_blks_total,
- s.param7 AS heap_blks_scanned,
- s.param8 AS index_rebuild_count
+ s.param5 AS heap_tuples_inserted,
+ s.param6 AS heap_tuples_updated,
+ s.param7 AS heap_tuples_deleted,
+ s.param8 AS heap_blks_total,
+ s.param9 AS heap_blks_scanned,
+ s.param10 AS index_rebuild_count
FROM (pg_stat_get_progress_info('REPACK'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_progress_vacuum| SELECT s.pid,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e3c1007abdf..cab34d68054 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -428,6 +428,7 @@ CatCacheHeader
CatalogId
CatalogIdMapEntry
CatalogIndexState
+ChangeContext
ChangeVarNodes_callback
ChangeVarNodes_context
ChannelName
@@ -505,6 +506,7 @@ CompressFileHandle
CompressionLocation
CompressorState
ComputeXidHorizonsResult
+ConcurrentChangeKind
ConditionVariable
ConditionVariableMinimallyPadded
ConditionalStack
@@ -646,6 +648,8 @@ DeclareCursorStmt
DecodedBkpBlock
DecodedXLogRecord
DecodingOutputState
+DecodingWorker
+DecodingWorkerShared
DefElem
DefElemAction
DefaultACLInfo
@@ -1305,6 +1309,7 @@ IndexElem
IndexFetchHeapData
IndexFetchTableData
IndexInfo
+IndexInsertState
IndexList
IndexOnlyScan
IndexOnlyScanState
@@ -2619,6 +2624,7 @@ ReorderBufferTupleCidKey
ReorderBufferUpdateProgressTxnCB
ReorderTuple
RepackCommand
+RepackDecodingState
RepackStmt
ReparameterizeForeignPathByChild_function
ReplOriginId
--
2.47.3
--mdsv5bdgfjjj6d77
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v46-0003-Use-BulkInsertState-when-copying-data-to-the-new.patch"
^ permalink raw reply [nested|flat] 18+ messages in thread
end of thread, other threads:[~2026-03-11 14:16 UTC | newest]
Thread overview: 18+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-01-06 21:55 Re: RFC: logical publication via inheritance root? Jacob Champion <[email protected]>
2023-01-09 08:41 ` Aleksander Alekseev <[email protected]>
2023-01-10 19:36 ` Jacob Champion <[email protected]>
2023-01-20 17:53 ` Jacob Champion <[email protected]>
2023-02-28 22:47 ` Jacob Champion <[email protected]>
2023-03-07 10:40 ` Aleksander Alekseev <[email protected]>
2023-03-08 22:07 ` Jacob Champion <[email protected]>
2023-03-31 22:17 ` Peter Smith <[email protected]>
2023-03-31 23:35 ` Jacob Champion <[email protected]>
2023-04-03 13:13 ` Aleksander Alekseev <[email protected]>
2023-06-16 13:25 ` Amit Kapila <[email protected]>
2023-04-04 03:53 ` Peter Smith <[email protected]>
2023-04-04 15:14 ` Jacob Champion <[email protected]>
2023-06-06 15:50 ` Jacob Champion <[email protected]>
2023-06-29 23:46 ` Jacob Champion <[email protected]>
2023-07-19 11:54 ` Aleksander Alekseev <[email protected]>
2023-08-31 18:16 ` Jacob Champion <[email protected]>
2026-03-11 14:16 [PATCH v46 2/7] Add CONCURRENTLY option to REPACK command. Antonin Houska <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox