public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
87+ messages / 16 participants
[nested] [flat]
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 6019f37f91..c7e138c121 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -694,6 +694,10 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Wed_Jul_31_16_56_16_2019_668)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v16-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v19 1/3] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 1d5450758e..b80305b6c5 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -702,6 +702,11 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ if (defined $params{primary_slot_name})
+ {
+ $self->append_conf('postgresql.conf',
+ "primary_slot_name = $params{primary_slot_name}");
+ }
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node, $params{standby}) if $params{has_restoring};
return;
--
2.20.1
--d6Gm4EdcadzBjdND
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v19-0002-Add-WAL-relief-vent-for-replication-slots.patch"
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 6019f37f91..2c7a894914 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -680,11 +680,7 @@ sub init_from_backup
chmod(0700, $data_path);
# Base configuration for this node
- $self->append_conf(
- 'postgresql.conf',
- qq(
-port = $port
-));
+ $self->append_conf('postgresql.conf', qq(port = $port));
if ($use_tcp)
{
$self->append_conf('postgresql.conf', "listen_addresses = '$host'");
@@ -694,6 +690,11 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ $self->append_conf('postgresql.conf', qq(port = $port));
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if defined $params{primary_slot_name};
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.20.1
--MP_/68vU=6Ib.yoYOTdA.M=EVne
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v14-0004-TAP-test-for-the-slot-limit-feature.patch
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8a2c6fc122..daca2e0085 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -672,11 +672,11 @@ sub init_from_backup
chmod(0700, $data_path);
# Base configuration for this node
- $self->append_conf(
- 'postgresql.conf',
- qq(
-port = $port
-));
+ $self->append_conf('postgresql.conf', qq(port = $port));
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Fri_Feb_22_14_12_28_2019_076)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v13-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 6019f37f91..c7e138c121 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -694,6 +694,10 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Tue_Jul_30_21_30_45_2019_680)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v15-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v4] Rejigger code
@ 2019-08-09 17:23 Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Alvaro Herrera @ 2019-08-09 17:23 UTC (permalink / raw)
---
src/backend/optimizer/util/plancat.c | 59 +++++++++++++++--------
src/backend/partitioning/partprune.c | 70 ++++++++++------------------
2 files changed, 64 insertions(+), 65 deletions(-)
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 98e99481c6..cf1761401d 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -78,6 +78,9 @@ static void set_relation_partition_info(PlannerInfo *root, RelOptInfo *rel,
static PartitionScheme find_partition_scheme(PlannerInfo *root, Relation rel);
static void set_baserel_partition_key_exprs(Relation relation,
RelOptInfo *rel);
+static void set_baserel_partition_constraint(Relation relation,
+ RelOptInfo *rel);
+
/*
* get_relation_info -
@@ -1267,25 +1270,9 @@ get_relation_constraints(PlannerInfo *root,
*/
if (include_partition && relation->rd_rel->relispartition)
{
- List *pcqual = RelationGetPartitionQual(relation);
-
- if (pcqual)
- {
- /*
- * Run the partition quals through const-simplification similar to
- * check constraints. We skip canonicalize_qual, though, because
- * partition quals should be in canonical form already; also,
- * since the qual is in implicit-AND format, we'd have to
- * explicitly convert it to explicit-AND format and back again.
- */
- pcqual = (List *) eval_const_expressions(root, (Node *) pcqual);
-
- /* Fix Vars to have the desired varno */
- if (varno != 1)
- ChangeVarNodes((Node *) pcqual, 1, varno, 0);
-
- result = list_concat(result, pcqual);
- }
+ /* make sure rel->partition_qual is set */
+ set_baserel_partition_constraint(relation, rel);
+ result = list_concat(result, rel->partition_qual);
}
table_close(relation, NoLock);
@@ -2149,7 +2136,7 @@ set_relation_partition_info(PlannerInfo *root, RelOptInfo *rel,
rel->boundinfo = partdesc->boundinfo;
rel->nparts = partdesc->nparts;
set_baserel_partition_key_exprs(relation, rel);
- rel->partition_qual = RelationGetPartitionQual(relation);
+ set_baserel_partition_constraint(relation, rel);
}
/*
@@ -2324,3 +2311,35 @@ set_baserel_partition_key_exprs(Relation relation,
*/
rel->nullable_partexprs = (List **) palloc0(sizeof(List *) * partnatts);
}
+
+/*
+ * set_baserel_partition_constraint
+ *
+ * Builds the partition constraint for the given base relation and sets it
+ * in the given RelOptInfo. All Var nodes are restamped with the relid of the
+ * given relation.
+ */
+static void
+set_baserel_partition_constraint(Relation relation, RelOptInfo *rel)
+{
+ List *partconstr;
+
+ if (rel->partition_qual) /* already done */
+ return;
+
+ /*
+ * Run the partition quals through const-simplification similar to check
+ * constraints. We skip canonicalize_qual, though, because partition
+ * quals should be in canonical form already; also, since the qual is in
+ * implicit-AND format, we'd have to explicitly convert it to explicit-AND
+ * format and back again.
+ */
+ partconstr = RelationGetPartitionQual(relation);
+ if (partconstr)
+ {
+ partconstr = (List *) expression_planner((Expr *) partconstr);
+ if (rel->relid != 1)
+ ChangeVarNodes((Node *) partconstr, 1, rel->relid, 0);
+ rel->partition_qual = partconstr;
+ }
+}
diff --git a/src/backend/partitioning/partprune.c b/src/backend/partitioning/partprune.c
index a1bd4efd0b..f619171ace 100644
--- a/src/backend/partitioning/partprune.c
+++ b/src/backend/partitioning/partprune.c
@@ -619,31 +619,16 @@ gen_partprune_steps(RelOptInfo *rel, List *clauses, PartClauseTarget target,
context->target = target;
/*
- * For sub-partitioned tables there's a corner case where if the
- * sub-partitioned table shares any partition keys with its parent, then
- * it's possible that the partitioning hierarchy allows the parent
- * partition to only contain a narrower range of values than the
- * sub-partitioned table does. In this case it is possible that we'd
- * include partitions that could not possibly have any tuples matching
- * 'clauses'. The possibility of such a partition arrangement is perhaps
- * unlikely for non-default partitions, but it may be more likely in the
- * case of default partitions, so we'll add the parent partition table's
- * partition qual to the clause list in this case only. This may result
- * in the default partition being eliminated.
+ * If this partitioned table is in turn a partition, and it shares any
+ * partition keys with its parent, then it's possible that the hierarchy
+ * allows the parent a narrower range of values than some of its
+ * partitions (particularly the default one). This is normally not
+ * useful, but it can be to prune the default partition.
*/
- if (partition_bound_has_default(rel->boundinfo) &&
- rel->partition_qual != NIL)
+ if (partition_bound_has_default(rel->boundinfo) && rel->partition_qual)
{
- List *partqual = rel->partition_qual;
-
- partqual = (List *) expression_planner((Expr *) partqual);
-
- /* Fix Vars to have the desired varno */
- if (rel->relid != 1)
- ChangeVarNodes((Node *) partqual, 1, rel->relid, 0);
-
/* Make a copy to avoid modifying the passed-in List */
- clauses = list_concat_copy(clauses, partqual);
+ clauses = list_concat_copy(clauses, rel->partition_qual);
}
/* Down into the rabbit-hole. */
@@ -867,6 +852,24 @@ gen_partprune_steps_internal(GeneratePruningStepsContext *context,
List *result = NIL;
ListCell *lc;
+ /*
+ * If this partitioned relation has a default partition and is itself
+ * a partition (as evidenced by partition_qual being not NIL), we first
+ * check if the clauses contradict the partition constraint. If they do,
+ * there's no need to generate any steps as it'd already be proven that no
+ * partitions need to be scanned.
+ *
+ * This is a measure of last resort only to be used because the default
+ * partition cannot be pruned using the steps; regular pruning, which is
+ * cheaper, is sufficient when no default partition exists.
+ */
+ if (partition_bound_has_default(context->rel->boundinfo) &&
+ predicate_refuted_by(context->rel->partition_qual, clauses, false))
+ {
+ context->contradictory = true;
+ return NIL;
+ }
+
memset(keyclauses, 0, sizeof(keyclauses));
foreach(lc, clauses)
{
@@ -1019,29 +1022,6 @@ gen_partprune_steps_internal(GeneratePruningStepsContext *context,
*/
}
- /*
- * If the clause contradicts the partition constraint, mark the clause
- * as contradictory and we're done. This is particularly helpful to
- * prune the default partition.
- */
- if (context->rel->partition_qual)
- {
- List *partconstr;
-
- partconstr = (List *)
- expression_planner((Expr *) context->rel->partition_qual);
- if (context->rel->relid != 1)
- ChangeVarNodes((Node *) partconstr, 1,
- context->rel->relid, 0);
- if (predicate_refuted_by(partconstr,
- list_make1(clause),
- false))
- {
- context->contradictory = true;
- return NIL;
- }
- }
-
/*
* See if we can match this clause to any of the partition keys.
*/
--
2.17.1
--bp/iNruPH9dso1Pn--
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v31 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 84494c4b81..0424ab9a29 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3729,6 +3729,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3794,6 +3798,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3829,6 +3834,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3878,6 +3884,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4755,6 +4762,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4965,6 +5011,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5125,19 +5172,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5154,6 +5201,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index cb968d03ec..768d6e1c0b 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Fri__29_Mar_2024_23_47_00_+0900_KGpmmDOIs1266Ib1
Content-Type: text/x-diff;
name="v31-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v31-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 401baddbfc6..dc746bccc8e 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3844,6 +3844,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3909,6 +3913,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3957,6 +3962,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -4003,6 +4009,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4898,6 +4905,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5138,6 +5184,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5312,11 +5359,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5334,8 +5381,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM
Content-Type: text/x-diff;
name="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v28 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 4b295f8da5..354ea94777 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4737,6 +4744,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4947,6 +4993,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5107,19 +5154,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5136,6 +5183,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Thu__1_Jun_2023_23_59_09_+0900_/G5+8nG46.f1T42K
Content-Type: text/x-diff;
name="v28-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v28-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 13cb516752..aa1da7b279 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3730,6 +3730,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3795,6 +3799,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3830,6 +3835,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3879,6 +3885,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState state);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4725,6 +4732,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4933,6 +4979,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5093,19 +5140,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -5117,6 +5164,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 194e8d5bc1..2e9f1ac8c7 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -261,6 +261,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__22_Apr_2022_11_29_39_+0900_ZOAC7UMt5e8j1Nvx
Content-Type: text/x-diff;
name="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 1d9869973c0..e5dd915096d 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index d8890d2c74..c7a2bc3ed8 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3570,6 +3570,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3635,6 +3639,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3670,6 +3675,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4448,6 +4454,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4642,6 +4687,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4802,11 +4848,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4818,6 +4882,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 9ef7f6d768..0cfb0b3c5c 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -251,6 +251,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__29_Oct_2021_18_16_28_+0900_jlYRKjywLqhZ7oyk
Content-Type: text/x-diff;
name="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index d8890d2c74..c7a2bc3ed8 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3570,6 +3570,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3635,6 +3639,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3670,6 +3675,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4448,6 +4454,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4642,6 +4687,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4802,11 +4848,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4818,6 +4882,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 9ef7f6d768..0cfb0b3c5c 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -251,6 +251,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Thu__23_Sep_2021_04_57_30_+0900_b5pmgR1N8oMaMz.U
Content-Type: text/x-diff;
name="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v23 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index fc0a4b2fa7..049007a484 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3570,6 +3570,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3635,6 +3639,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3670,6 +3675,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4448,6 +4454,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4642,6 +4687,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4802,11 +4848,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4818,6 +4882,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 9ef7f6d768..0cfb0b3c5c 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -251,6 +251,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu
Content-Type: text/x-diff;
name="v23-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v23-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 1d9869973c0..e5dd915096d 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v32 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 84494c4b81..0424ab9a29 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3729,6 +3729,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3794,6 +3798,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3829,6 +3834,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3878,6 +3884,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4755,6 +4762,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4965,6 +5011,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5125,19 +5172,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5154,6 +5201,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index cb968d03ec..768d6e1c0b 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Sun__31_Mar_2024_22_59_31_+0900_msknEviJj08_wgqO
Content-Type: text/x-diff;
name="v32-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v32-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 1d9869973c0..e5dd915096d 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 401baddbfc6..dc746bccc8e 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3844,6 +3844,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3909,6 +3913,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3957,6 +3962,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -4003,6 +4009,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4898,6 +4905,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5138,6 +5184,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5312,11 +5359,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5334,8 +5381,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM
Content-Type: text/x-diff;
name="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 13cb516752..aa1da7b279 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3730,6 +3730,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3795,6 +3799,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3830,6 +3835,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3879,6 +3885,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState state);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4725,6 +4732,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4933,6 +4979,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5093,19 +5140,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -5117,6 +5164,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 194e8d5bc1..2e9f1ac8c7 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -261,6 +261,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__22_Apr_2022_14_58_01_+0900_MN3L/o2YUVF2g4zw
Content-Type: text/x-diff;
name="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 401baddbfc6..dc746bccc8e 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3844,6 +3844,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3909,6 +3913,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3957,6 +3962,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -4003,6 +4009,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4898,6 +4905,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5138,6 +5184,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5312,11 +5359,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5334,8 +5381,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM
Content-Type: text/x-diff;
name="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v26 04/10] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index e08bd9a370..adc853a570 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3651,6 +3651,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3716,6 +3720,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3751,6 +3756,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4539,6 +4545,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4733,6 +4778,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4893,11 +4939,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4909,6 +4973,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index e1271420e5..697a6f6d55 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -255,6 +255,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Mon__14_Mar_2022_19_12_17_+0900_E9HVp8j5QFkIIek.
Content-Type: text/x-diff;
name="v26-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Disposition: attachment;
filename="v26-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v25 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 1a9c1ac290..0fcb1aa0b5 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3651,6 +3651,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3716,6 +3720,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3751,6 +3756,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4529,6 +4535,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4723,6 +4768,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4883,11 +4929,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4899,6 +4963,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index e1271420e5..697a6f6d55 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -255,6 +255,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__4_Feb_2022_01_48_06_+0900_N8BNZpfOR27sWrgY
Content-Type: text/x-diff;
name="v25-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Disposition: attachment;
filename="v25-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
@ 2022-03-23 18:50 ` Peter Eisentraut <[email protected]>
6 siblings, 0 replies; 87+ messages in thread
From: Peter Eisentraut @ 2022-03-23 18:50 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; Euler Taveira <[email protected]>; [email protected]
On 18.03.22 23:34, Andrew Dunstan wrote:
> On 3/15/22 09:51, Peter Eisentraut wrote:
>> On 21.02.22 13:09, Euler Taveira wrote:
>>> A new tool called pg_subscriber does this conversion and is tightly
>>> integrated
>>> with Postgres.
>>
>> Are we comfortable with the name pg_subscriber? It seems too general.
>> Are we planning other subscriber-related operations in the future? If
>> so, we should at least make this one use a --create option or
>> something like that.
>
>
> Not really sold on the name (and I didn't much like the name
> pglogical_create_subscriber either, although it's a cool facility and
> I'm happy to see us adopting something like it).
>
> ISTM we should have a name that conveys that we are *converting* a
> replica or equivalent to a subscriber.
The pglogical tool includes the pg_basebackup run, so it actually
"creates" the subscriber from scratch. Whether this tool is also doing
that is still being discussed.
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
@ 2022-03-29 12:34 ` Peter Eisentraut <[email protected]>
2022-08-02 18:56 ` Re: speed up a logical replica setup Jacob Champion <[email protected]>
6 siblings, 1 reply; 87+ messages in thread
From: Peter Eisentraut @ 2022-03-29 12:34 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Euler Taveira <[email protected]>; +Cc: [email protected]
On 22.03.22 02:25, Andres Freund wrote:
> On 2022-02-21 09:09:12 -0300, Euler Taveira wrote:
>> A new tool called pg_subscriber does this conversion and is tightly integrated
>> with Postgres.
>
> Given that this has been submitted just before the last CF and is a patch of
> nontrivial size, has't made significant progress ISTM it should be moved to
> the next CF?
done
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2022-03-29 12:34 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2022-08-02 18:56 ` Jacob Champion <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Jacob Champion @ 2022-08-02 18:56 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Euler Taveira <[email protected]>; +Cc: [email protected]
This entry has been waiting on author input for a while (our current
threshold is roughly two weeks), so I've marked it Returned with
Feedback.
Once you think the patchset is ready for review again, you (or any
interested party) can resurrect the patch entry by visiting
https://commitfest.postgresql.org/38/3556/
and changing the status to "Needs Review", and then changing the
status again to "Move to next CF". (Don't forget the second step;
hopefully we will have streamlined this in the near future!)
Thanks,
--Jacob
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
@ 2024-01-08 07:05 ` Hayato Kuroda (Fujitsu) <[email protected]>
6 siblings, 0 replies; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-08 07:05 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>
Dear Amit,
> On Fri, Jan 5, 2024 at 3:36 PM Hayato Kuroda (Fujitsu)
> <[email protected]> wrote:
> >
> > I love your proposal, so I want to join the review. Here are my first comments.
> >
> > 01.
> > Should we restrict that `--subscriber-conninfo` must not have hostname or IP?
> > We want users to execute pg_subscriber on the target, right?
> >
>
> I don't see any harm in users giving those information but we should
> have some checks to ensure that the server is in standby mode and is
> running locally. The other related point is do we need to take input
> for the target cluster directory from the user? Can't we fetch that
> information once we are connected to standby?
I think that functions like inet_client_addr() may be able to use, but it returns
NULL only when the connection is via a Unix-domain socket. Can we restrict
pg_subscriber to use such a socket?
> >
> > 05.
> > I found that the connection string for each subscriptions have a setting
> > "fallback_application_name=pg_subscriber". Can we remove it?
> >
> > ```
> > postgres=# SELECT subconninfo FROM pg_subscription;
> > subconninfo
> >
> ---------------------------------------------------------------------------------
> > user=postgres port=5431 fallback_application_name=pg_subscriber
> dbname=postgres
> > (1 row)
> > ```
>
> Can that help distinguish the pg_subscriber connection on the publisher?
>
Note that this connection string is used between the publisher instance and the
subscriber instance (not pg_subscriber client application). Also, the
fallback_application_name would be replaced to the name of subscriber in
run_apply_worker()->walrcv_connect(). Actually the value would not be used.
See below output on publisher.
```
publisher=# SELECT application_name, backend_type FROM pg_stat_activity where backend_type = 'walsender';
application_name | backend_type
----------------------+--------------
pg_subscriber_5_9411 | walsender
(1 row)
```
Or, if you mean to say that this can distinguish whether the subscription is used
by pg_subscriber or not. I think it is sufficient the current format of name.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
@ 2024-01-18 09:18 ` Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-31 13:03 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
6 siblings, 2 replies; 87+ messages in thread
From: Peter Eisentraut @ 2024-01-18 09:18 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On 11.01.24 23:15, Euler Taveira wrote:
> A new tool called pg_subscriber can convert a physical replica into a
> logical replica. It runs on the target server and should be able to
> connect to the source server (publisher) and the target server (subscriber).
Can we have a discussion on the name?
I find the name pg_subscriber too general.
The replication/backup/recovery tools in PostgreSQL are usually named
along the lines of "verb - object". (Otherwise, they would all be
called "pg_backup"??) Moreover, "pg_subscriber" also sounds like the
name of the program that runs the subscriber itself, like what the
walreceiver does now.
Very early in this thread, someone mentioned the name
pg_create_subscriber, and of course there is pglogical_create_subscriber
as the historical predecessor. Something along those lines seems better
to me. Maybe there are other ideas.
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-18 09:37 ` Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
1 sibling, 1 reply; 87+ messages in thread
From: Amit Kapila @ 2024-01-18 09:37 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Thu, Jan 18, 2024 at 2:49 PM Peter Eisentraut <[email protected]> wrote:
>
> On 11.01.24 23:15, Euler Taveira wrote:
> > A new tool called pg_subscriber can convert a physical replica into a
> > logical replica. It runs on the target server and should be able to
> > connect to the source server (publisher) and the target server (subscriber).
>
> Can we have a discussion on the name?
>
> I find the name pg_subscriber too general.
>
> The replication/backup/recovery tools in PostgreSQL are usually named
> along the lines of "verb - object". (Otherwise, they would all be
> called "pg_backup"??) Moreover, "pg_subscriber" also sounds like the
> name of the program that runs the subscriber itself, like what the
> walreceiver does now.
>
> Very early in this thread, someone mentioned the name
> pg_create_subscriber, and of course there is pglogical_create_subscriber
> as the historical predecessor. Something along those lines seems better
> to me. Maybe there are other ideas.
>
The other option could be pg_createsubscriber on the lines of
pg_verifybackup and pg_combinebackup. Yet other options could be
pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
sounds like we are doing some work to create the subscriber which I
think is the case here.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
@ 2024-01-22 06:55 ` Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Peter Eisentraut @ 2024-01-22 06:55 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On 18.01.24 10:37, Amit Kapila wrote:
> The other option could be pg_createsubscriber on the lines of
> pg_verifybackup and pg_combinebackup.
Yes, that spelling would be more consistent.
> Yet other options could be
> pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> sounds like we are doing some work to create the subscriber which I
> think is the case here.
I see your point here. pg_createsubscriber is not like createuser in
that it just runs an SQL command. It does something different than
CREATE SUBSCRIBER. So a different verb would make that clearer. Maybe
something from here: https://www.thesaurus.com/browse/convert
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-22 09:08 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-22 09:08 UTC (permalink / raw)
To: 'Peter Eisentraut' <[email protected]>; Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
Dear Peter,
>
> > Yet other options could be
> > pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> > sounds like we are doing some work to create the subscriber which I
> > think is the case here.
>
> I see your point here. pg_createsubscriber is not like createuser in
> that it just runs an SQL command. It does something different than
> CREATE SUBSCRIBER. So a different verb would make that clearer. Maybe
> something from here: https://www.thesaurus.com/browse/convert
I read the link and found a good verb "switch". So, how about using "pg_switchsubscriber"?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-22 09:22 ` Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Amit Kapila @ 2024-01-22 09:22 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Euler Taveira <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Mon, Jan 22, 2024 at 2:38 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
> >
> > > Yet other options could be
> > > pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> > > sounds like we are doing some work to create the subscriber which I
> > > think is the case here.
> >
> > I see your point here. pg_createsubscriber is not like createuser in
> > that it just runs an SQL command. It does something different than
> > CREATE SUBSCRIBER.
Right.
So a different verb would make that clearer. Maybe
> > something from here: https://www.thesaurus.com/browse/convert
>
> I read the link and found a good verb "switch". So, how about using "pg_switchsubscriber"?
>
I also initially thought on these lines and came up with a name like
pg_convertsubscriber but didn't feel strongly about it as that would
have sounded meaningful if we use a name like
pg_convertstandbytosubscriber. Now, that has become too long. Having
said that, I am not opposed to it having a name on those lines. BTW,
another option that occurred to me today is pg_preparesubscriber. We
internally create slots and then wait for wal, etc. which makes me
sound like adding 'prepare' in the name can also explain the purpose.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
@ 2024-01-23 23:44 ` Euler Taveira <[email protected]>
2024-01-25 09:27 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Euler Taveira @ 2024-01-23 23:44 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; [email protected] <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Mon, Jan 22, 2024, at 6:22 AM, Amit Kapila wrote:
> On Mon, Jan 22, 2024 at 2:38 PM Hayato Kuroda (Fujitsu)
> <[email protected]> wrote:
> > >
> > > > Yet other options could be
> > > > pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> > > > sounds like we are doing some work to create the subscriber which I
> > > > think is the case here.
> > >
> > > I see your point here. pg_createsubscriber is not like createuser in
> > > that it just runs an SQL command. It does something different than
> > > CREATE SUBSCRIBER.
>
> Right.
Subscriber has a different meaning of subscription. Subscription is an SQL
object. Subscriber is the server (node in replication terminology) where the
subscription resides. Having said that pg_createsubscriber doesn't seem a bad
name because you are creating a new subscriber. (Indeed, you are transforming /
converting but "create" seems closer and users can infer that it is a tool to
build a new logical replica.
> So a different verb would make that clearer. Maybe
> > > something from here: https://www.thesaurus.com/browse/convert
> >
> > I read the link and found a good verb "switch". So, how about using "pg_switchsubscriber"?
> >
>
> I also initially thought on these lines and came up with a name like
> pg_convertsubscriber but didn't feel strongly about it as that would
> have sounded meaningful if we use a name like
> pg_convertstandbytosubscriber. Now, that has become too long. Having
> said that, I am not opposed to it having a name on those lines. BTW,
> another option that occurred to me today is pg_preparesubscriber. We
> internally create slots and then wait for wal, etc. which makes me
> sound like adding 'prepare' in the name can also explain the purpose.
I think "convert" and "transform" fit for this case. However, "create",
"convert" and "transform" have 6, 7 and 9 characters, respectively. I suggest
that we avoid long names (subscriber already has 10 characters). My preference
is pg_createsubscriber.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-25 09:27 ` Peter Eisentraut <[email protected]>
2024-01-25 10:16 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Peter Eisentraut @ 2024-01-25 09:27 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; Amit Kapila <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On 24.01.24 00:44, Euler Taveira wrote:
> Subscriber has a different meaning of subscription. Subscription is an SQL
> object. Subscriber is the server (node in replication terminology) where the
> subscription resides. Having said that pg_createsubscriber doesn't seem
> a bad
> name because you are creating a new subscriber. (Indeed, you are
> transforming /
> converting but "create" seems closer and users can infer that it is a
> tool to
> build a new logical replica.
That makes sense.
(Also, the problem with "convert" etc. is that "convertsubscriber" would
imply that you are converting an existing subscriber to something else.
It would need to be something like "convertbackup" then, which doesn't
seem helpful.)
> I think "convert" and "transform" fit for this case. However, "create",
> "convert" and "transform" have 6, 7 and 9 characters, respectively. I
> suggest
> that we avoid long names (subscriber already has 10 characters). My
> preference
> is pg_createsubscriber.
That seems best to me.
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 09:27 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-25 10:16 ` Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-25 10:16 UTC (permalink / raw)
To: 'Peter Eisentraut' <[email protected]>; Euler Taveira <[email protected]>; Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
Dear Peter,
Thanks for giving your idea!
> > Subscriber has a different meaning of subscription. Subscription is an SQL
> > object. Subscriber is the server (node in replication terminology) where the
> > subscription resides. Having said that pg_createsubscriber doesn't seem
> > a bad
> > name because you are creating a new subscriber. (Indeed, you are
> > transforming /
> > converting but "create" seems closer and users can infer that it is a
> > tool to
> > build a new logical replica.
>
> That makes sense.
>
> (Also, the problem with "convert" etc. is that "convertsubscriber" would
> imply that you are converting an existing subscriber to something else.
> It would need to be something like "convertbackup" then, which doesn't
> seem helpful.)
>
> > I think "convert" and "transform" fit for this case. However, "create",
> > "convert" and "transform" have 6, 7 and 9 characters, respectively. I
> > suggest
> > that we avoid long names (subscriber already has 10 characters). My
> > preference
> > is pg_createsubscriber.
>
> That seems best to me.
Just FYI - I'm ok to change the name to pg_createsubscriber.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-31 13:03 ` Fabrízio de Royes Mello <[email protected]>
1 sibling, 0 replies; 87+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 13:03 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Thu, Jan 18, 2024 at 6:19 AM Peter Eisentraut <[email protected]>
wrote:
>
> Very early in this thread, someone mentioned the name
> pg_create_subscriber, and of course there is pglogical_create_subscriber
> as the historical predecessor. Something along those lines seems better
> to me. Maybe there are other ideas.
>
I've mentioned it upthread because of this pet project [1] that is one of
the motivations behind upstream this facility.
[1] https://github.com/fabriziomello/pg_create_subscriber
--
Fabrízio de Royes Mello
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
@ 2024-01-18 09:43 ` Amit Kapila <[email protected]>
6 siblings, 0 replies; 87+ messages in thread
From: Amit Kapila @ 2024-01-18 09:43 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Euler Taveira <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Tue, Jan 16, 2024 at 11:58 AM Shubham Khanna
<[email protected]> wrote:
>
> On Thu, Dec 21, 2023 at 11:47 AM Amit Kapila <[email protected]> wrote:
> >
> >
> > 4. Can we see some numbers with various sizes of databases (cluster)
> > to see how it impacts the time for small to large-size databases as
> > compared to the traditional method? This might help us with giving
> > users advice on when to use this tool. We can do this bit later as
> > well when the patch is closer to being ready for commit.
>
> I have done the Performance testing and attached the results to
> compare the 'Execution Time' between 'logical replication' and
> 'pg_subscriber' for 100MB, 1GB and 5GB data:
> | 100MB | 1GB | 5GB
> Logical rep (2 w) | 1.815s | 14.895s | 75.541s
> Logical rep (4 w) | 1.194s | 9.484s | 46.938s
> Logical rep (8 w) | 0.828s | 6.422s | 31.704s
> Logical rep(10 w)| 0.646s | 3.843s | 18.425s
> pg_subscriber | 3.977s | 9.988s | 12.665s
>
> Here, 'w' stands for 'workers'. I have included the tests to see the
> test result variations with different values for
> 'max_sync_workers_per_subscription' ranging from 2 to 10. I ran the
> tests for different data records; for 100MB I put 3,00,000 Records,
> for 1GB I put 30,00,000 Records and for 5GB I put 1,50,00,000 Records.
> It is observed that 'pg_subscriber' is better when the table size is
> more.
>
Thanks for the tests. IIUC, it shows for smaller data this tool can
take more time. Can we do perf to see if there is something we can do
about reducing the overhead?
> Next I plan to run these tests for 10GB and 20GB to see if this trend
> continues or not.
>
Okay, that makes sense.
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
@ 2024-01-19 08:48 ` Shubham Khanna <[email protected]>
2024-01-25 05:54 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
6 siblings, 1 reply; 87+ messages in thread
From: Shubham Khanna @ 2024-01-19 08:48 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Tue, Jan 16, 2024 at 11:58 AM Shubham Khanna
<[email protected]> wrote:
>
> On Thu, Dec 21, 2023 at 11:47 AM Amit Kapila <[email protected]> wrote:
> >
> > On Wed, Dec 6, 2023 at 12:53 PM Euler Taveira <[email protected]> wrote:
> > >
> > > On Thu, Nov 9, 2023, at 8:12 PM, Michael Paquier wrote:
> > >
> > > On Thu, Nov 09, 2023 at 03:41:53PM +0100, Peter Eisentraut wrote:
> > > > On 08.11.23 00:12, Michael Paquier wrote:
> > > >> - Should the subdirectory pg_basebackup be renamed into something more
> > > >> generic at this point? All these things are frontend tools that deal
> > > >> in some way with the replication protocol to do their work. Say
> > > >> a replication_tools?
> > > >
> > > > Seems like unnecessary churn. Nobody has complained about any of the other
> > > > tools in there.
> > >
> > > Not sure. We rename things across releases in the tree from time to
> > > time, and here that's straight-forward.
> > >
> > >
> > > Based on this discussion it seems we have a consensus that this tool should be
> > > in the pg_basebackup directory. (If/when we agree with the directory renaming,
> > > it could be done in a separate patch.) Besides this move, the v3 provides a dry
> > > run mode. It basically executes every routine but skip when should do
> > > modifications. It is an useful option to check if you will be able to run it
> > > without having issues with connectivity, permission, and existing objects
> > > (replication slots, publications, subscriptions). Tests were slightly improved.
> > > Messages were changed to *not* provide INFO messages by default and --verbose
> > > provides INFO messages and --verbose --verbose also provides DEBUG messages. I
> > > also refactored the connect_database() function into which the connection will
> > > always use the logical replication mode. A bug was fixed in the transient
> > > replication slot name. Ashutosh review [1] was included. The code was also indented.
> > >
> > > There are a few suggestions from Ashutosh [2] that I will reply in another
> > > email.
> > >
> > > I'm still planning to work on the following points:
> > >
> > > 1. improve the cleanup routine to point out leftover objects if there is any
> > > connection issue.
> > >
> >
> > I think this is an important part. Shall we try to write to some file
> > the pending objects to be cleaned up? We do something like that during
> > the upgrade.
> >
> > > 2. remove the physical replication slot if the standby is using one
> > > (primary_slot_name).
> > > 3. provide instructions to promote the logical replica into primary, I mean,
> > > stop the replication between the nodes and remove the replication setup
> > > (publications, subscriptions, replication slots). Or even include another
> > > action to do it. We could add both too.
> > >
> > > Point 1 should be done. Points 2 and 3 aren't essential but will provide a nice
> > > UI for users that would like to use it.
> > >
> >
> > Isn't point 2 also essential because how would otherwise such a slot
> > be advanced or removed?
> >
> > A few other points:
> > ==============
> > 1. Previously, I asked whether we need an additional replication slot
> > patch created to get consistent LSN and I see the following comment in
> > the patch:
> >
> > + *
> > + * XXX we should probably use the last created replication slot to get a
> > + * consistent LSN but it should be changed after adding pg_basebackup
> > + * support.
> >
> > Yeah, sure, we may want to do that after backup support and we can
> > keep a comment for the same but I feel as the patch stands today,
> > there is no good reason to keep it. Also, is there a reason that we
> > can't create the slots after backup is complete and before we write
> > recovery parameters
> >
> > 2.
> > + appendPQExpBuffer(str,
> > + "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
> > + "WITH (create_slot = false, copy_data = false, enabled = false)",
> > + dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
> >
> > Shouldn't we enable two_phase by default for newly created
> > subscriptions? Is there a reason for not doing so?
> >
> > 3. How about sync slots on the physical standby if present? Do we want
> > to retain those as it is or do we need to remove those? We are
> > actively working on the patch [1] for the same.
> >
> > 4. Can we see some numbers with various sizes of databases (cluster)
> > to see how it impacts the time for small to large-size databases as
> > compared to the traditional method? This might help us with giving
> > users advice on when to use this tool. We can do this bit later as
> > well when the patch is closer to being ready for commit.
>
> I have done the Performance testing and attached the results to
> compare the 'Execution Time' between 'logical replication' and
> 'pg_subscriber' for 100MB, 1GB and 5GB data:
> | 100MB | 1GB | 5GB
> Logical rep (2 w) | 1.815s | 14.895s | 75.541s
> Logical rep (4 w) | 1.194s | 9.484s | 46.938s
> Logical rep (8 w) | 0.828s | 6.422s | 31.704s
> Logical rep(10 w)| 0.646s | 3.843s | 18.425s
> pg_subscriber | 3.977s | 9.988s | 12.665s
>
> Here, 'w' stands for 'workers'. I have included the tests to see the
> test result variations with different values for
> 'max_sync_workers_per_subscription' ranging from 2 to 10. I ran the
> tests for different data records; for 100MB I put 3,00,000 Records,
> for 1GB I put 30,00,000 Records and for 5GB I put 1,50,00,000 Records.
> It is observed that 'pg_subscriber' is better when the table size is
> more.
> Next I plan to run these tests for 10GB and 20GB to see if this trend
> continues or not.
I have done the Performance testing and attached the results to
compare the 'Execution Time' between 'logical replication' and
'pg_subscriber' for 10GB and 20GB data:
| 10GB | 20GB
Logical rep (2 w) | 157.131s| 343.191s
Logical rep (4 w) | 116.627s| 240.480s
Logical rep (8 w) | 95.237s | 275.715s
Logical rep(10 w)| 92.792s | 280.538s
pg_subscriber | 22.734s | 25.661s
As expected, we can see that pg_subscriber is very much better in
ideal cases with approximately 7x times better in case of 10GB and 13x
times better in case of 20GB.
I'm attaching the script files which have the details of the test
scripts used and the excel file has the test run details. The
'pg_subscriber.pl' file is for 'Streaming Replication' and the
'logical_replication.pl' file is for 'Logical Replication'.
Note: For 20GB the record count should be changed to 6,00,00,000 and
the 'max_sync_workers_per_subscription' needs to be adjusted for
different logical replication tests with different workers.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] logical_replication.pl (2.8K, ../../CAHv8RjKYo1Xdkj6WhFZ32Mg4-+9EuNgau=JZH8U7_NPv2gFmGQ@mail.gmail.com/2-logical_replication.pl)
download
[application/octet-stream] pg_subscriber.pl (3.3K, ../../CAHv8RjKYo1Xdkj6WhFZ32Mg4-+9EuNgau=JZH8U7_NPv2gFmGQ@mail.gmail.com/3-pg_subscriber.pl)
download
[application/vnd.openxmlformats-officedocument.spreadsheetml.sheet] time_stamp(comparison).xlsx (11.5K, ../../CAHv8RjKYo1Xdkj6WhFZ32Mg4-+9EuNgau=JZH8U7_NPv2gFmGQ@mail.gmail.com/4-time_stamp%28comparison%29.xlsx)
download
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-19 08:48 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
@ 2024-01-25 05:54 ` Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-25 05:54 UTC (permalink / raw)
To: 'Shubham Khanna' <[email protected]>; Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
Dear hackers,
Based on the requirement, I have profiled the performance test. It showed bottlenecks
are in small-data case are mainly two - starting a server and waiting until the
recovery is done.
# Tested source code
V7 patch set was applied atop HEAD(0eb23285). No configure options were specified
when it was built.
# Tested workload
I focused on only 100MB/1GB cases because bigger ones have already had good performance.
(Number of inserted tuples were same as previous tests)
I used bash script instead of tap test framework. See attached. Executed SQLs and
operations were almost the same.
As you can see, I tested only one-db case. Results may be changed if the number
of databases were changed.
# Measurement
Some debug logs which output current time were added (please see diff file).
I picked up some events and done at before/after them. Below bullets showed the measured ones:
* Starting a server
* Stopping a server
* Creating replication slots
* Creating publications
* Waiting until the recovery ended
* Creating subscriptions
# Result
Below table shows the elapsed time for these events. Raw data is also available
by the attached excel file.
|Event category |100MB case [sec]|1GB [sec]|
|Starting a server |1.414 |1.417 |
|Stoping a server |0.506 |0.506 |
|Creating replication slots |0.005 |0.007 |
|Creating publications |0.001 |0.002 |
|Waiting until the recovery ended|1.603 |14.529 |
|Creating subscriptions |0.012 |0.012 |
|Total |3.541 |16.473 |
|actual time |4.37 |17.271 |
As you can see, starting servers and waiting seem slow. We cannot omit these,
but setting smaller shared_buffers will reduce the start time. One approach is
to overwrite the GUC to smaller value, but I think we cannot determine the
appropriate value.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] run.sh (2.2K, ../../TY3PR01MB9889C1E33C73DC064B3AEA90F57A2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-run.sh)
download
[application/vnd.openxmlformats-officedocument.spreadsheetml.sheet] perf_result.xlsx (17.3K, ../../TY3PR01MB9889C1E33C73DC064B3AEA90F57A2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-perf_result.xlsx)
download
[application/octet-stream] add_debug_log.diff (8.2K, ../../TY3PR01MB9889C1E33C73DC064B3AEA90F57A2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-add_debug_log.diff)
download | inline diff:
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 355738c..290be7c 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -675,13 +675,66 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a replication slot %s start: %s", replslotname, timebuf);
+ }
+
/* Create replication slot on publisher. */
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a replication slot %s end: %s", replslotname, timebuf);
+ }
+
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a publication start: %s", timebuf);
+ }
+
/* Also create a publication */
create_publication(conn, primary, perdb);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a publication end: %s", timebuf);
+ }
+
disconnect_database(conn);
}
@@ -839,6 +892,19 @@ wait_for_end_recovery(const char *base_conninfo, const char *dbname)
if (conn == NULL)
exit(1);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a wait_for start: %s", timebuf);
+ }
+
for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -875,6 +941,19 @@ wait_for_end_recovery(const char *base_conninfo, const char *dbname)
pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a wait_for stop: %s", timebuf);
+ }
+
disconnect_database(conn);
/*
@@ -1059,6 +1138,19 @@ create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
pg_log_debug("command is: %s", str->data);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a subscription %s start: %s", subname, timebuf);
+ }
+
if (!dry_run)
{
res = PQexec(conn, str->data);
@@ -1071,6 +1163,19 @@ create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
}
}
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a subscription %s end: %s", subname, timebuf);
+ }
+
/* for cleanup purposes */
perdb->made_subscription = true;
@@ -1277,8 +1382,35 @@ start_standby_server(StandbyInfo *standby, unsigned short subport,
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
standby->bindir,
standby->pgdata, subport, server_start_log);
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl start begin: %s", timebuf);
+ }
+
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl start end: %s", timebuf);
+ }
}
static char *
@@ -1723,8 +1855,38 @@ main(int argc, char **argv)
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
standby.bindir, standby.pgdata);
+
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop begin: %s", timebuf);
+ }
+
+
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop end: %s", timebuf);
+ }
}
/*
@@ -1756,6 +1918,7 @@ main(int argc, char **argv)
consistent_lsn = create_logical_replication_slot(conn, true,
&dbarr.perdb[0]);
+
/*
* Write recovery parameters.
*
@@ -1856,9 +2019,36 @@ main(int argc, char **argv)
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
standby.bindir, standby.pgdata);
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop for subscriber begin: %s", timebuf);
+ }
+
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop for subscriber end: %s", timebuf);
+ }
+
/*
* Change system identifier.
*/
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
@ 2024-01-22 07:06 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 23:58 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
6 siblings, 2 replies; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-22 07:06 UTC (permalink / raw)
To: '[email protected]' <[email protected]>; +Cc: 'Shlok Kyal' <[email protected]>; 'vignesh C' <[email protected]>; 'Michael Paquier' <[email protected]>; 'Peter Eisentraut' <[email protected]>; 'Andres Freund' <[email protected]>; 'Ashutosh Bapat' <[email protected]>; 'Amit Kapila' <[email protected]>; 'Euler Taveira' <[email protected]>
Dear hackers,
>
> 15.
> I found that subscriptions cannot be started if tuples are inserted on publisher
> after creating temp_replslot. After starting a subscriber, I got below output on the
> log.
>
> ```
> ERROR: could not receive data from WAL stream: ERROR: publication
> "pg_subscriber_5" does not exist
> CONTEXT: slot "pg_subscriber_5_3632", output plugin "pgoutput", in the change
> callback, associated LSN 0/30008A8
> LOG: background worker "logical replication apply worker" (PID 3669) exited
> with exit code 1
> ```
>
> But this is strange. I confirmed that the specified publication surely exists.
> Do you know the reason?
>
> ```
> publisher=# SELECT pubname FROM pg_publication;
> pubname
> -----------------
> pg_subscriber_5
> (1 row)
> ```
>
I analyzed and found a reason. This is because publications are invisible for some transactions.
As the first place, below operations were executed in this case.
Tuples were inserted after getting consistent_lsn, but before starting the standby.
After doing the workload, I confirmed again that the publication was created.
1. on primary, logical replication slots were created.
2. on primary, another replication slot was created.
3. ===on primary, some tuples were inserted. ===
4. on standby, a server process was started
5. on standby, the process waited until all changes have come.
6. on primary, publications were created.
7. on standby, subscriptions were created.
8. on standby, a replication progress for each subscriptions was set to given LSN (got at step2).
=====pg_subscriber finished here=====
9. on standby, a server process was started again
10. on standby, subscriptions were enabled. They referred slots created at step1.
11. on primary, decoding was started but ERROR was raised.
In this case, tuples were inserted *before creating publication*.
So I thought that the decoded transaction could not see the publication because
it was committed after insertions.
One solution is to create a publication before creating a consistent slot.
Changes which came before creating the slot were surely replicated to the standby,
so upcoming transactions can see the object. We are planning to patch set to fix
the issue in this approach.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-22 09:30 ` Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
1 sibling, 2 replies; 87+ messages in thread
From: Shlok Kyal @ 2024-01-22 09:30 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
Dear Euler, hackers,
We fixed some of the comments posted in the thread. We have created it
as top-up patch 0002 and 0003.
0002 patch contains the following changes:
* Add a timeout option for the recovery option, per [1]. The code was
basically ported from pg_ctl.c.
* Reject if the target server is not a standby, per [2]
* Raise FATAL error if --subscriber-conninfo specifies non-local server, per [3]
(not sure it is really needed, so feel free reject the part.)
* Add check for max_replication_slots and wal_level; as per [4]
* Add -u and -p options; as per [5]
* Addressed comment except 5 and 8 in [6] and comment in [7]
0003 patch contains fix for bug reported in [8].
Feel free to merge parts of 0002 and 0003 if it looks good to you.
Thanks Kuroda-san to make patch 0003 and a part of patch 0002.
[1]: https://www.postgresql.org/message-id/CANhcyEUCt-g4JLQU3Q3ofFk_Vt-Tqh3ZdXoLcpT8fjz9LY_-ww%40mail.gma...
[2]: https://www.postgresql.org/message-id/CANhcyEUCt-g4JLQU3Q3ofFk_Vt-Tqh3ZdXoLcpT8fjz9LY_-ww%40mail.gma...
[3]: https://www.postgresql.org/message-id/TY3PR01MB98895BA6C1D72CB8582CACC4F5682%40TY3PR01MB9889.jpnprd0...
[4]: https://www.postgresql.org/message-id/CALDaNm098Jkbh%2Bye6zMj9Ro9j1bBe6FfPV80BFbs1%3DpUuTJ07g%40mail...
[5]: https://www.postgresql.org/message-id/CAA4eK1JB_ko7a5JMS3WfAn583RadAKCDhiE9JgmfMA8ZZ5xcQw%40mail.gma...
[6]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[7]: https://www.postgresql.org/message-id/CANhcyEXjGmryoZPACS_i-joqvcz5e6Zb3u4g38SAy_iSTGhShg%40mail.gma...
[8]: https://www.postgresql.org/message-id/TY3PR01MB9889C5D55206DDD978627D07F5752%40TY3PR01MB9889.jpnprd0...
Thanks and regards
Shlok Kyal
Attachments:
[application/octet-stream] v6-0003-Fix-publication-does-not-exist-error.patch (1.6K, ../../CANhcyEXhpfWgc0-FdHmUG6WGD=2cxYqBePr1yxJ8VSZ0gV3XKA@mail.gmail.com/2-v6-0003-Fix-publication-does-not-exist-error.patch)
download | inline diff:
From 5d2b49e55888cdc36a38208d58cf16a5960821dc Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:36:20 +0530
Subject: [PATCH v6 3/3] Fix publication does not exist error.
Fix publication does not exist error.
---
src/bin/pg_basebackup/pg_subscriber.c | 23 +++--------------------
1 file changed, 3 insertions(+), 20 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 0dc87e919b..8b1a92b68b 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -677,6 +677,9 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ /* Also create a publication */
+ create_publication(conn, primary, perdb);
+
disconnect_database(conn);
}
@@ -1792,26 +1795,6 @@ main(int argc, char **argv)
*/
wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
- /*
- * Create a publication for each database. This step should be executed
- * after promoting the subscriber to avoid replicating unnecessary
- * objects.
- */
- for (i = 0; i < dbarr.ndbs; i++)
- {
- LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
-
- /* Connect to publisher. */
- conn = connect_database(primary.base_conninfo, perdb->dbname);
- if (conn == NULL)
- exit(1);
-
- /* Also create a publication */
- create_publication(conn, &primary, perdb);
-
- disconnect_database(conn);
- }
-
/*
* Create a subscription for each database.
*/
--
2.34.1
[application/octet-stream] v6-0002-Address-some-comments-proposed-on-hackers.patch (58.3K, ../../CANhcyEXhpfWgc0-FdHmUG6WGD=2cxYqBePr1yxJ8VSZ0gV3XKA@mail.gmail.com/3-v6-0002-Address-some-comments-proposed-on-hackers.patch)
download | inline diff:
From ad645b61dad1a8ce03ab0ad28a0c44d0a943cc3e Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:42:34 +0530
Subject: [PATCH v6 2/3] Address some comments proposed on -hackers
The patch has following changes:
* Some comments reported on the thread
* Add a timeout option for the recovery option
* Reject if the target server is not a standby
* Reject when the --subscriber-conninfo specifies non-local server
* Add -u and -p options
* Check wal_level and max_replication_slot parameters
---
doc/src/sgml/ref/pg_subscriber.sgml | 21 +-
src/bin/pg_basebackup/pg_subscriber.c | 911 +++++++++++-------
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 9 +-
.../t/041_pg_subscriber_standby.pl | 8 +-
4 files changed, 600 insertions(+), 349 deletions(-)
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
index 553185c35f..eaabfc7053 100644
--- a/doc/src/sgml/ref/pg_subscriber.sgml
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -16,12 +16,18 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_subscriber</refname>
- <refpurpose>create a new logical replica from a standby server</refpurpose>
+ <refpurpose>Convert a standby replica to a logical replica</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_subscriber</command>
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><replaceable>datadir</replaceable></arg>
+ <arg choice="plain"><option>-P</option>
+ <replaceable>publisher-conninfo</replaceable></arg>
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><replaceable>subscriber-conninfo</replaceable></arg>
<arg rep="repeat"><replaceable>option</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
@@ -29,17 +35,18 @@ PostgreSQL documentation
<refsect1>
<title>Description</title>
<para>
- <application>pg_subscriber</application> takes the publisher and subscriber
- connection strings, a cluster directory from a standby server and a list of
- database names and it sets up a new logical replica using the physical
- recovery process.
+ pg_subscriber creates a new <link
+ linkend="logical-replication-subscription">subscriber</link> from a physical
+ standby server. This allows users to quickly set up logical replication
+ system.
</para>
<para>
- The <application>pg_subscriber</application> should be run at the target
+ The <application>pg_subscriber</application> has to be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
+ The target server should accept logical replication connection from
+ localhost.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index e998c29f9e..0dc87e919b 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1,12 +1,12 @@
/*-------------------------------------------------------------------------
*
* pg_subscriber.c
- * Create a new logical replica from a standby server
+ * Convert a standby replica to a logical replica
*
* Copyright (C) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/bin/pg_subscriber/pg_subscriber.c
+ * src/bin/pg_basebackup/pg_subscriber.c
*
*-------------------------------------------------------------------------
*/
@@ -32,81 +32,122 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
-static bool get_exec_path(const char *path);
+static char *get_base_conninfo(char *conninfo, char *dbname);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_control_for_standby(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static char *use_primary_slot_name(void);
-static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb);
+static bool create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void modify_sysid(const char *bindir, const char *datadir);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void wait_for_end_recovery(const char *base_conninfo,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ char *base_conninfo,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log);
+static char *construct_sub_conninfo(char *username, unsigned short subport);
#define USEC_PER_SEC 1000000
-#define WAIT_INTERVAL 1 /* 1 second */
+#define DEFAULT_WAIT 60
+#define WAITS_PER_SEC 10 /* should divide USEC_PER_SEC evenly */
+#define DEF_PGSPORT 50111
/* Options */
-static const char *progname;
-
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
-static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
+static int wait_seconds = DEFAULT_WAIT;
+static bool retain = false;
static bool dry_run = false;
static bool success = false;
+static const char *progname;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
+enum PGSWaitPMResult
+{
+ PGS_POSTMASTER_READY,
+ PGS_POSTMASTER_STANDBY,
+ PGS_POSTMASTER_STILL_STARTING,
+ PGS_POSTMASTER_FAILED
+};
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
-enum WaitPMResult
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
{
- POSTMASTER_READY,
- POSTMASTER_STANDBY,
- POSTMASTER_STILL_STARTING,
- POSTMASTER_FAILED
-};
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
/*
@@ -125,41 +166,39 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
disconnect_database(conn);
}
}
}
-
- if (made_transient_replslot)
- {
- conn = connect_database(dbinfo[0].pubconninfo);
- if (conn != NULL)
- {
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
- disconnect_database(conn);
- }
- }
}
static void
@@ -184,17 +223,16 @@ usage(void)
/*
* Validate a connection string. Returns a base connection string that is a
- * connection string without a database name plus a fallback application name.
- * Since we might process multiple databases, each database name will be
- * appended to this base connection string to provide a final connection string.
- * If the second argument (dbname) is not null, returns dbname if the provided
- * connection string contains it. If option --database is not provided, uses
- * dbname as the only database to setup the logical replica.
- * It is the caller's responsibility to free the returned connection string and
- * dbname.
+ * connection string without a database name. Since we might process multiple
+ * databases, each database name will be appended to this base connection
+ * string to provide a final connection string. If the second argument (dbname)
+ * is not null, returns dbname if the provided connection string contains it.
+ * If option --database is not provided, uses dbname as the only database to
+ * setup the logical replica. It is the caller's responsibility to free the
+ * returned connection string and dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -203,7 +241,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on publisher");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -231,10 +269,6 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
}
- if (i > 0)
- appendPQExpBufferChar(buf, ' ');
- appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
-
ret = pg_strdup(buf->data);
destroyPQExpBuffer(buf);
@@ -244,15 +278,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -277,30 +312,10 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
-
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ /* Extract the directory part from the path */
+ Assert(p = strrchr(pg_ctl_path, 'p'));
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -364,49 +379,36 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
SimpleStringListCell *cell;
int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ *perdb = (LogicalRepPerdbInfo *) pg_malloc0(sizeof(LogicalRepPerdbInfo) *
+ ndbs);
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -424,6 +426,9 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
+
return conn;
}
@@ -436,19 +441,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -471,43 +475,39 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_control_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -516,7 +516,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -551,7 +551,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -571,14 +571,15 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* Return a palloc'd slot name if the replication is using one.
*/
static char *
-use_primary_slot_name(void)
+use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb)
{
PGconn *conn;
PGresult *res;
PQExpBuffer str = createPQExpBuffer();
char *slot_name;
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -604,7 +605,7 @@ use_primary_slot_name(void)
disconnect_database(conn);
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -634,17 +635,19 @@ use_primary_slot_name(void)
}
static bool
-create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr)
{
int i;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr->ndbs; i++)
{
PGconn *conn;
PGresult *res;
char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -664,27 +667,14 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
disconnect_database(conn);
@@ -701,30 +691,36 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (!temporary)
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name, NAMEDATALEN);
+ else
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+
+ if(temporary)
+ appendPQExpBufferStr(str, " TEMPORARY");
+
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
pg_log_debug("command is: %s", str->data);
@@ -734,17 +730,14 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
- else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -758,14 +751,15 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -775,7 +769,7 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -825,19 +819,22 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* Returns after the server finishes the recovery process.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(const char *base_conninfo, const char *dbname)
{
PGconn *conn;
PGresult *res;
- int status = POSTMASTER_STILL_STARTING;
+ int status = PGS_POSTMASTER_STILL_STARTING;
+ int cnt;
+ int rc;
+ char *pg_ctl_cmd;
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(base_conninfo, dbname);
if (conn == NULL)
exit(1);
- for (;;)
+ for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -865,17 +862,32 @@ wait_for_end_recovery(const char *conninfo)
*/
if (!in_recovery || dry_run)
{
- status = POSTMASTER_READY;
+ status = PGS_POSTMASTER_READY;
break;
}
/* Keep waiting. */
- pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
disconnect_database(conn);
- if (status == POSTMASTER_STILL_STARTING)
+ /*
+ * If timeout is reached exit the pg_subscriber and stop the standby node.
+ */
+ if (cnt >= wait_seconds * WAITS_PER_SEC)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ if (status == PGS_POSTMASTER_STILL_STARTING)
{
pg_log_error("server did not end recovery");
exit(1);
@@ -888,17 +900,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -918,7 +934,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -931,7 +947,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -942,9 +958,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -954,14 +970,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -973,16 +989,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
@@ -990,7 +1009,7 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1011,19 +1030,27 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname, concat_conninfo_dbname(base_conninfo, perdb->dbname), pubname);
pg_log_debug("command is: %s", str->data);
@@ -1033,14 +1060,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1052,16 +1079,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", subname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1069,7 +1099,7 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1088,18 +1118,21 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1140,7 +1173,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1154,7 +1187,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1173,16 +1206,20 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1191,7 +1228,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1203,6 +1240,61 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+static void
+start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ int rc;
+ char *pg_ctl_cmd;
+
+ if (server_start_log[0] == '\0')
+ {
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
+ standby->bindir,
+ standby->pgdata, subport, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static char *
+construct_sub_conninfo(char *username, unsigned short subport)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ if (username)
+ appendPQExpBuffer(buf, "user=%s ", username);
+
+ appendPQExpBuffer(buf, "port=%d fallback_application_name=%s",
+ subport, progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
int
main(int argc, char **argv)
{
@@ -1214,6 +1306,10 @@ main(int argc, char **argv)
{"publisher-conninfo", required_argument, NULL, 'P'},
{"subscriber-conninfo", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
+ {"timeout", required_argument, NULL, 't'},
+ {"username", required_argument, NULL, 'u'},
+ {"port", required_argument, NULL, 'p'},
+ {"retain", no_argument, NULL, 'r'},
{"dry-run", no_argument, NULL, 'n'},
{"verbose", no_argument, NULL, 'v'},
{NULL, 0, NULL, 0}
@@ -1225,20 +1321,15 @@ main(int argc, char **argv)
char *pg_ctl_cmd;
- char *base_dir;
- char *server_start_log;
-
- char timebuf[128];
- struct timeval time;
- time_t tt;
+ char base_dir[MAXPGPATH];
+ char server_start_log[MAXPGPATH] = {0};
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
+ unsigned short subport = DEF_PGSPORT;
+ char *username = NULL;
+
struct stat statbuf;
PGconn *conn;
@@ -1250,6 +1341,13 @@ main(int argc, char **argv)
int i;
+ PGresult *res;
+
+ char *wal_level;
+ int max_replication_slots;
+ int nslots_old;
+ int nslots_new;
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
progname = get_progname(argv[0]);
@@ -1286,28 +1384,40 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:t:u:p:rnv",
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
+ canonicalize_path(standby.pgdata);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
break;
- case 'S':
- sub_conninfo_str = pg_strdup(optarg);
- break;
case 'd':
/* Ignore duplicated database names. */
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
+ case 't':
+ wait_seconds = atoi(optarg);
+ break;
+ case 'u':
+ pfree(username);
+ username = pg_strdup(optarg);
+ break;
+ case 'p':
+ if ((subport = atoi(optarg)) <= 0)
+ pg_fatal("invalid old port number");
+ break;
+ case 'r':
+ retain = true;
+ break;
case 'n':
dry_run = true;
break;
@@ -1335,7 +1445,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1358,21 +1468,14 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
- if (sub_conninfo_str == NULL)
- {
- pg_log_error("no subscriber connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo);
+ if (primary.base_conninfo == NULL)
exit(1);
+ standby.base_conninfo = construct_sub_conninfo(username, subport);
+
if (database_names.head == NULL)
{
pg_log_info("no database was specified");
@@ -1385,7 +1488,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1399,25 +1502,25 @@ main(int argc, char **argv)
}
/*
- * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ * Get the absolute path of binaries on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_control_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1426,8 +1529,8 @@ main(int argc, char **argv)
/*
* Create the output directory to store any data generated by this tool.
*/
- base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s",
+ standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1441,7 +1544,153 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid",
+ standby.pgdata);
+
+ /* Start the standby server anyway */
+ start_standby_server(&standby, subport, server_start_log);
+
+ /*
+ * Check wal_level in publisher and the max_replication_slots of publisher
+ */
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name IN ('wal_level', 'max_replication_slots') "
+ "ORDER BY name DESC;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters on publisher");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 2)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ wal_level = PQgetvalue(res, 0, 0);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("wal_level must be \"logical\", but is set to \"%s\"", wal_level);
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 1, 0));
+ nslots_new = nslots_old + dbarr.ndbs + 1;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ conn = connect_database(standby.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Check the max_replication_slots in subscriber
+ */
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots on subscriber");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on subscriber");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'max_replication_slots';");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 0, 0));
+ nslots_new = nslots_old + dbarr.ndbs;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+
+ /*
+ * Exit the pg_subscriber if the node is not a standby server.
+ */
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ /* Check if the server is in recovery */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") != 0)
+ {
+ pg_log_error("pg_subscriber is supported only on standby server");
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* Stop the subscriber if it is a standby server. Before executing the
@@ -1457,14 +1706,18 @@ main(int argc, char **argv)
* replication slot has no use after the transformation, hence, it
* will be removed at the end of this process.
*/
- primary_slot_name = use_primary_slot_name();
- if (primary_slot_name != NULL)
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ standby.primary_slot_name = use_primary_slot_name(&primary,
+ &standby,
+ &dbarr.perdb[0]);
+ if (standby.primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
pg_log_info("subscriber is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1472,7 +1725,7 @@ main(int argc, char **argv)
/*
* Create a replication slot for each database on the publisher.
*/
- if (!create_all_logical_replication_slots(dbinfo))
+ if (!create_all_logical_replication_slots(&primary, &dbarr))
exit(1);
/*
@@ -1492,11 +1745,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true,
+ &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1522,7 +1775,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1532,54 +1785,29 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby_server(&standby, subport, server_start_log);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
/*
* Create a publication for each database. This step should be executed
* after promoting the subscriber to avoid replicating unnecessary
* objects.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- char pubname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
/* Connect to publisher. */
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
- create_publication(conn, &dbinfo[i]);
+ /* Also create a publication */
+ create_publication(conn, &primary, perdb);
disconnect_database(conn);
}
@@ -1587,20 +1815,25 @@ main(int argc, char **argv)
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
+
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, primary.base_conninfo, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
+
+ drop_publication(conn, perdb);
disconnect_database(conn);
}
@@ -1613,19 +1846,21 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
- pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
disconnect_database(conn);
}
@@ -1634,20 +1869,22 @@ main(int argc, char **argv)
*/
pg_log_info("stopping the subscriber");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* Remove log file generated by this tool, if it runs successfully.
* Otherwise, file is kept that may provide useful debugging information.
*/
- unlink(server_start_log);
+ if (!retain)
+ unlink(server_start_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
index 4ebff76b2d..9915b8cb3c 100644
--- a/src/bin/pg_basebackup/t/040_pg_subscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -37,8 +37,13 @@ command_fails(
'--verbose',
'--pgdata', $datadir,
'--publisher-conninfo', 'dbname=postgres',
- '--subscriber-conninfo', 'dbname=postgres'
],
'no database name specified');
-
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ ],
+ 'subscriber connection string specnfied non-local server');
done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
index fbcd0fc82b..4e26607611 100644
--- a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -51,25 +51,27 @@ $node_s->start;
$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
$node_p->wait_for_replay_catchup($node_s);
+$node_f->stop;
+
# Run pg_subscriber on about-to-fail node F
command_fails(
[
'pg_subscriber', '--verbose',
'--pgdata', $node_f->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
'subscriber data directory is not a copy of the source database cluster');
+$node_s->stop;
+
# dry run mode on node S
command_ok(
[
'pg_subscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
@@ -82,6 +84,7 @@ $node_s->start;
# Check if node S is still a standby
is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
+$node_s->stop;
# Run pg_subscriber on node S
command_ok(
@@ -89,7 +92,6 @@ command_ok(
'pg_subscriber', '--verbose',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
--
2.34.1
[application/octet-stream] v6-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (69.2K, ../../CANhcyEXhpfWgc0-FdHmUG6WGD=2cxYqBePr1yxJ8VSZ0gV3XKA@mail.gmail.com/4-v6-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From 7b808c5a927e3abf98b1e3bb62ec64dd5b80b013 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v6 1/3] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 284 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1657 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2153 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..553185c35f
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,284 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>create a new logical replica from a standby server</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a standby server and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a standby
+ server.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a standby server. Stop the standby server if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a standby server at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..e998c29f9e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1657 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_subscriber/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static char *use_primary_slot_name(void);
+static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name plus a fallback application name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Return a palloc'd slot name if the replication is using one.
+ */
+static char *
+use_primary_slot_name(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+ char *slot_name;
+
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings WHERE name = 'primary_slot_name'");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain parameter information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ /*
+ * If primary_slot_name is an empty string, the current replication
+ * connection is not using a replication slot, bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "") == 0)
+ {
+ PQclear(res);
+ return NULL;
+ }
+
+ slot_name = pg_strdup(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots r INNER JOIN pg_stat_activity a ON (r.active_pid = a.pid) WHERE slot_name = '%s'", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ return NULL;
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ return slot_name;
+}
+
+static bool
+create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+{
+ int i;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ PGconn *conn;
+ PGresult *res;
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * Stop the subscriber if it is a standby server. Before executing the
+ * transformation steps, make sure the subscriber is not running because
+ * one of the steps is to modify some recovery parameters that require a
+ * restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Since the standby server is running, check if it is using an
+ * existing replication slot for WAL retention purposes. This
+ * replication slot has no use after the transformation, hence, it
+ * will be removed at the end of this process.
+ */
+ primary_slot_name = use_primary_slot_name();
+ if (primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+
+ pg_log_info("subscriber is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+
+ /*
+ * Create a replication slot for each database on the publisher.
+ */
+ if (!create_all_logical_replication_slots(dbinfo))
+ exit(1);
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a publication for each database. This step should be executed
+ * after promoting the subscriber to avoid replicating unnecessary
+ * objects.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+
+ /* Connect to publisher. */
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ create_publication(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * Remove log file generated by this tool, if it runs successfully.
+ * Otherwise, file is kept that may provide useful debugging information.
+ */
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
@ 2024-01-23 02:11 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
1 sibling, 1 reply; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-23 02:11 UTC (permalink / raw)
To: 'Shlok Kyal' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
Dear hackers,
> We fixed some of the comments posted in the thread. We have created it
> as top-up patch 0002 and 0003.
I found that the CFbot raised an ERROR. Also, it may not work well in case
of production build.
PSA the fixed patch set.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v7-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (67.0K, ../../TY3PR01MB98890AB22AECA2F503EB7858F5742@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v7-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From 773b6d187892a7e0aea0edf1547b86ad2b2c8e2f Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v7 1/3] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 284 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1657 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2153 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..553185c35f
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,284 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>create a new logical replica from a standby server</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a standby server and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a standby
+ server.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a standby server. Stop the standby server if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a standby server at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..e998c29f9e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1657 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_subscriber/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static char *use_primary_slot_name(void);
+static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name plus a fallback application name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Return a palloc'd slot name if the replication is using one.
+ */
+static char *
+use_primary_slot_name(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+ char *slot_name;
+
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings WHERE name = 'primary_slot_name'");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain parameter information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ /*
+ * If primary_slot_name is an empty string, the current replication
+ * connection is not using a replication slot, bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "") == 0)
+ {
+ PQclear(res);
+ return NULL;
+ }
+
+ slot_name = pg_strdup(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots r INNER JOIN pg_stat_activity a ON (r.active_pid = a.pid) WHERE slot_name = '%s'", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ return NULL;
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ return slot_name;
+}
+
+static bool
+create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+{
+ int i;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ PGconn *conn;
+ PGresult *res;
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * Stop the subscriber if it is a standby server. Before executing the
+ * transformation steps, make sure the subscriber is not running because
+ * one of the steps is to modify some recovery parameters that require a
+ * restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Since the standby server is running, check if it is using an
+ * existing replication slot for WAL retention purposes. This
+ * replication slot has no use after the transformation, hence, it
+ * will be removed at the end of this process.
+ */
+ primary_slot_name = use_primary_slot_name();
+ if (primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+
+ pg_log_info("subscriber is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+
+ /*
+ * Create a replication slot for each database on the publisher.
+ */
+ if (!create_all_logical_replication_slots(dbinfo))
+ exit(1);
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a publication for each database. This step should be executed
+ * after promoting the subscriber to avoid replicating unnecessary
+ * objects.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+
+ /* Connect to publisher. */
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ create_publication(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * Remove log file generated by this tool, if it runs successfully.
+ * Otherwise, file is kept that may provide useful debugging information.
+ */
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.43.0
[application/octet-stream] v7-0002-Address-some-comments-proposed-on-hackers.patch (56.5K, ../../TY3PR01MB98890AB22AECA2F503EB7858F5742@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v7-0002-Address-some-comments-proposed-on-hackers.patch)
download | inline diff:
From cfb77f4c599417527f7bfbcb7e8d90a4b09b5108 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:42:34 +0530
Subject: [PATCH v7 2/3] Address some comments proposed on -hackers
The patch has following changes:
* Some comments reported on the thread
* Add a timeout option for the recovery option
* Reject if the target server is not a standby
* Reject when the --subscriber-conninfo specifies non-local server
* Add -u and -p options
* Check wal_level and max_replication_slot parameters
---
doc/src/sgml/ref/pg_subscriber.sgml | 21 +-
src/bin/pg_basebackup/pg_subscriber.c | 911 +++++++++++-------
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 9 +-
.../t/041_pg_subscriber_standby.pl | 8 +-
4 files changed, 601 insertions(+), 348 deletions(-)
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
index 553185c35f..eaabfc7053 100644
--- a/doc/src/sgml/ref/pg_subscriber.sgml
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -16,12 +16,18 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_subscriber</refname>
- <refpurpose>create a new logical replica from a standby server</refpurpose>
+ <refpurpose>Convert a standby replica to a logical replica</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_subscriber</command>
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><replaceable>datadir</replaceable></arg>
+ <arg choice="plain"><option>-P</option>
+ <replaceable>publisher-conninfo</replaceable></arg>
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><replaceable>subscriber-conninfo</replaceable></arg>
<arg rep="repeat"><replaceable>option</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
@@ -29,17 +35,18 @@ PostgreSQL documentation
<refsect1>
<title>Description</title>
<para>
- <application>pg_subscriber</application> takes the publisher and subscriber
- connection strings, a cluster directory from a standby server and a list of
- database names and it sets up a new logical replica using the physical
- recovery process.
+ pg_subscriber creates a new <link
+ linkend="logical-replication-subscription">subscriber</link> from a physical
+ standby server. This allows users to quickly set up logical replication
+ system.
</para>
<para>
- The <application>pg_subscriber</application> should be run at the target
+ The <application>pg_subscriber</application> has to be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
+ The target server should accept logical replication connection from
+ localhost.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index e998c29f9e..3880d15ef9 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1,12 +1,12 @@
/*-------------------------------------------------------------------------
*
* pg_subscriber.c
- * Create a new logical replica from a standby server
+ * Convert a standby replica to a logical replica
*
* Copyright (C) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/bin/pg_subscriber/pg_subscriber.c
+ * src/bin/pg_basebackup/pg_subscriber.c
*
*-------------------------------------------------------------------------
*/
@@ -32,81 +32,122 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
-static bool get_exec_path(const char *path);
+static char *get_base_conninfo(char *conninfo, char *dbname);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_control_for_standby(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static char *use_primary_slot_name(void);
-static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb);
+static bool create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void modify_sysid(const char *bindir, const char *datadir);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void wait_for_end_recovery(const char *base_conninfo,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ char *base_conninfo,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log);
+static char *construct_sub_conninfo(char *username, unsigned short subport);
#define USEC_PER_SEC 1000000
-#define WAIT_INTERVAL 1 /* 1 second */
+#define DEFAULT_WAIT 60
+#define WAITS_PER_SEC 10 /* should divide USEC_PER_SEC evenly */
+#define DEF_PGSPORT 50111
/* Options */
-static const char *progname;
-
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
-static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
+static int wait_seconds = DEFAULT_WAIT;
+static bool retain = false;
static bool dry_run = false;
static bool success = false;
+static const char *progname;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
+enum PGSWaitPMResult
+{
+ PGS_POSTMASTER_READY,
+ PGS_POSTMASTER_STANDBY,
+ PGS_POSTMASTER_STILL_STARTING,
+ PGS_POSTMASTER_FAILED
+};
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
-enum WaitPMResult
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
{
- POSTMASTER_READY,
- POSTMASTER_STANDBY,
- POSTMASTER_STILL_STARTING,
- POSTMASTER_FAILED
-};
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
/*
@@ -125,41 +166,39 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
disconnect_database(conn);
}
}
}
-
- if (made_transient_replslot)
- {
- conn = connect_database(dbinfo[0].pubconninfo);
- if (conn != NULL)
- {
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
- disconnect_database(conn);
- }
- }
}
static void
@@ -184,17 +223,16 @@ usage(void)
/*
* Validate a connection string. Returns a base connection string that is a
- * connection string without a database name plus a fallback application name.
- * Since we might process multiple databases, each database name will be
- * appended to this base connection string to provide a final connection string.
- * If the second argument (dbname) is not null, returns dbname if the provided
- * connection string contains it. If option --database is not provided, uses
- * dbname as the only database to setup the logical replica.
- * It is the caller's responsibility to free the returned connection string and
- * dbname.
+ * connection string without a database name. Since we might process multiple
+ * databases, each database name will be appended to this base connection
+ * string to provide a final connection string. If the second argument (dbname)
+ * is not null, returns dbname if the provided connection string contains it.
+ * If option --database is not provided, uses dbname as the only database to
+ * setup the logical replica. It is the caller's responsibility to free the
+ * returned connection string and dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -203,7 +241,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on publisher");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -231,10 +269,6 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
}
- if (i > 0)
- appendPQExpBufferChar(buf, ' ');
- appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
-
ret = pg_strdup(buf->data);
destroyPQExpBuffer(buf);
@@ -244,15 +278,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -277,30 +312,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -364,49 +381,36 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
SimpleStringListCell *cell;
int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ *perdb = (LogicalRepPerdbInfo *) pg_malloc0(sizeof(LogicalRepPerdbInfo) *
+ ndbs);
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -424,6 +428,9 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
+
return conn;
}
@@ -436,19 +443,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -471,43 +477,39 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_control_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -516,7 +518,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -551,7 +553,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -571,14 +573,15 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* Return a palloc'd slot name if the replication is using one.
*/
static char *
-use_primary_slot_name(void)
+use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb)
{
PGconn *conn;
PGresult *res;
PQExpBuffer str = createPQExpBuffer();
char *slot_name;
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -604,7 +607,7 @@ use_primary_slot_name(void)
disconnect_database(conn);
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -634,17 +637,19 @@ use_primary_slot_name(void)
}
static bool
-create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr)
{
int i;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr->ndbs; i++)
{
PGconn *conn;
PGresult *res;
char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -664,27 +669,14 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
disconnect_database(conn);
@@ -701,30 +693,36 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (!temporary)
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name, NAMEDATALEN);
+ else
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+
+ if(temporary)
+ appendPQExpBufferStr(str, " TEMPORARY");
+
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
pg_log_debug("command is: %s", str->data);
@@ -734,17 +732,14 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
- else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -758,14 +753,15 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -775,7 +771,7 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -825,19 +821,22 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* Returns after the server finishes the recovery process.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(const char *base_conninfo, const char *dbname)
{
PGconn *conn;
PGresult *res;
- int status = POSTMASTER_STILL_STARTING;
+ int status = PGS_POSTMASTER_STILL_STARTING;
+ int cnt;
+ int rc;
+ char *pg_ctl_cmd;
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(base_conninfo, dbname);
if (conn == NULL)
exit(1);
- for (;;)
+ for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -865,17 +864,32 @@ wait_for_end_recovery(const char *conninfo)
*/
if (!in_recovery || dry_run)
{
- status = POSTMASTER_READY;
+ status = PGS_POSTMASTER_READY;
break;
}
/* Keep waiting. */
- pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
disconnect_database(conn);
- if (status == POSTMASTER_STILL_STARTING)
+ /*
+ * If timeout is reached exit the pg_subscriber and stop the standby node.
+ */
+ if (cnt >= wait_seconds * WAITS_PER_SEC)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ if (status == PGS_POSTMASTER_STILL_STARTING)
{
pg_log_error("server did not end recovery");
exit(1);
@@ -888,17 +902,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -918,7 +936,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -931,7 +949,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -942,9 +960,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -954,14 +972,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -973,16 +991,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
@@ -990,7 +1011,7 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1011,19 +1032,27 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname, concat_conninfo_dbname(base_conninfo, perdb->dbname), pubname);
pg_log_debug("command is: %s", str->data);
@@ -1033,14 +1062,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1052,16 +1081,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1069,7 +1101,7 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1088,18 +1120,21 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1140,7 +1175,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1154,7 +1189,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1173,16 +1208,20 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1191,7 +1230,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1203,6 +1242,61 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+static void
+start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ int rc;
+ char *pg_ctl_cmd;
+
+ if (server_start_log[0] == '\0')
+ {
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
+ standby->bindir,
+ standby->pgdata, subport, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static char *
+construct_sub_conninfo(char *username, unsigned short subport)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ if (username)
+ appendPQExpBuffer(buf, "user=%s ", username);
+
+ appendPQExpBuffer(buf, "port=%d fallback_application_name=%s",
+ subport, progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
int
main(int argc, char **argv)
{
@@ -1214,6 +1308,10 @@ main(int argc, char **argv)
{"publisher-conninfo", required_argument, NULL, 'P'},
{"subscriber-conninfo", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
+ {"timeout", required_argument, NULL, 't'},
+ {"username", required_argument, NULL, 'u'},
+ {"port", required_argument, NULL, 'p'},
+ {"retain", no_argument, NULL, 'r'},
{"dry-run", no_argument, NULL, 'n'},
{"verbose", no_argument, NULL, 'v'},
{NULL, 0, NULL, 0}
@@ -1225,20 +1323,15 @@ main(int argc, char **argv)
char *pg_ctl_cmd;
- char *base_dir;
- char *server_start_log;
-
- char timebuf[128];
- struct timeval time;
- time_t tt;
+ char base_dir[MAXPGPATH];
+ char server_start_log[MAXPGPATH] = {0};
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
+ unsigned short subport = DEF_PGSPORT;
+ char *username = NULL;
+
struct stat statbuf;
PGconn *conn;
@@ -1250,6 +1343,13 @@ main(int argc, char **argv)
int i;
+ PGresult *res;
+
+ char *wal_level;
+ int max_replication_slots;
+ int nslots_old;
+ int nslots_new;
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
progname = get_progname(argv[0]);
@@ -1286,28 +1386,40 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:t:u:p:rnv",
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
+ canonicalize_path(standby.pgdata);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
break;
- case 'S':
- sub_conninfo_str = pg_strdup(optarg);
- break;
case 'd':
/* Ignore duplicated database names. */
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
+ case 't':
+ wait_seconds = atoi(optarg);
+ break;
+ case 'u':
+ pfree(username);
+ username = pg_strdup(optarg);
+ break;
+ case 'p':
+ if ((subport = atoi(optarg)) <= 0)
+ pg_fatal("invalid old port number");
+ break;
+ case 'r':
+ retain = true;
+ break;
case 'n':
dry_run = true;
break;
@@ -1335,7 +1447,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1358,21 +1470,14 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
- if (sub_conninfo_str == NULL)
- {
- pg_log_error("no subscriber connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo);
+ if (primary.base_conninfo == NULL)
exit(1);
+ standby.base_conninfo = construct_sub_conninfo(username, subport);
+
if (database_names.head == NULL)
{
pg_log_info("no database was specified");
@@ -1385,7 +1490,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1399,25 +1504,25 @@ main(int argc, char **argv)
}
/*
- * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ * Get the absolute path of binaries on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_control_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1426,8 +1531,8 @@ main(int argc, char **argv)
/*
* Create the output directory to store any data generated by this tool.
*/
- base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s",
+ standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1441,7 +1546,153 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid",
+ standby.pgdata);
+
+ /* Start the standby server anyway */
+ start_standby_server(&standby, subport, server_start_log);
+
+ /*
+ * Check wal_level in publisher and the max_replication_slots of publisher
+ */
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name IN ('wal_level', 'max_replication_slots') "
+ "ORDER BY name DESC;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters on publisher");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 2)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ wal_level = PQgetvalue(res, 0, 0);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("wal_level must be \"logical\", but is set to \"%s\"", wal_level);
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 1, 0));
+ nslots_new = nslots_old + dbarr.ndbs + 1;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ conn = connect_database(standby.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Check the max_replication_slots in subscriber
+ */
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots on subscriber");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on subscriber");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'max_replication_slots';");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 0, 0));
+ nslots_new = nslots_old + dbarr.ndbs;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+
+ /*
+ * Exit the pg_subscriber if the node is not a standby server.
+ */
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ /* Check if the server is in recovery */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") != 0)
+ {
+ pg_log_error("pg_subscriber is supported only on standby server");
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* Stop the subscriber if it is a standby server. Before executing the
@@ -1457,14 +1708,18 @@ main(int argc, char **argv)
* replication slot has no use after the transformation, hence, it
* will be removed at the end of this process.
*/
- primary_slot_name = use_primary_slot_name();
- if (primary_slot_name != NULL)
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ standby.primary_slot_name = use_primary_slot_name(&primary,
+ &standby,
+ &dbarr.perdb[0]);
+ if (standby.primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
pg_log_info("subscriber is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1472,7 +1727,7 @@ main(int argc, char **argv)
/*
* Create a replication slot for each database on the publisher.
*/
- if (!create_all_logical_replication_slots(dbinfo))
+ if (!create_all_logical_replication_slots(&primary, &dbarr))
exit(1);
/*
@@ -1492,11 +1747,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true,
+ &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1522,7 +1777,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1532,54 +1787,29 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby_server(&standby, subport, server_start_log);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
/*
* Create a publication for each database. This step should be executed
* after promoting the subscriber to avoid replicating unnecessary
* objects.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- char pubname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
/* Connect to publisher. */
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
- create_publication(conn, &dbinfo[i]);
+ /* Also create a publication */
+ create_publication(conn, &primary, perdb);
disconnect_database(conn);
}
@@ -1587,20 +1817,25 @@ main(int argc, char **argv)
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
+
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, primary.base_conninfo, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
+
+ drop_publication(conn, perdb);
disconnect_database(conn);
}
@@ -1613,19 +1848,21 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
- pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
disconnect_database(conn);
}
@@ -1634,20 +1871,22 @@ main(int argc, char **argv)
*/
pg_log_info("stopping the subscriber");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* Remove log file generated by this tool, if it runs successfully.
* Otherwise, file is kept that may provide useful debugging information.
*/
- unlink(server_start_log);
+ if (!retain)
+ unlink(server_start_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
index 4ebff76b2d..9915b8cb3c 100644
--- a/src/bin/pg_basebackup/t/040_pg_subscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -37,8 +37,13 @@ command_fails(
'--verbose',
'--pgdata', $datadir,
'--publisher-conninfo', 'dbname=postgres',
- '--subscriber-conninfo', 'dbname=postgres'
],
'no database name specified');
-
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ ],
+ 'subscriber connection string specnfied non-local server');
done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
index fbcd0fc82b..4e26607611 100644
--- a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -51,25 +51,27 @@ $node_s->start;
$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
$node_p->wait_for_replay_catchup($node_s);
+$node_f->stop;
+
# Run pg_subscriber on about-to-fail node F
command_fails(
[
'pg_subscriber', '--verbose',
'--pgdata', $node_f->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
'subscriber data directory is not a copy of the source database cluster');
+$node_s->stop;
+
# dry run mode on node S
command_ok(
[
'pg_subscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
@@ -82,6 +84,7 @@ $node_s->start;
# Check if node S is still a standby
is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
+$node_s->stop;
# Run pg_subscriber on node S
command_ok(
@@ -89,7 +92,6 @@ command_ok(
'pg_subscriber', '--verbose',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
--
2.43.0
[application/octet-stream] v7-0003-Fix-publication-does-not-exist-error.patch (1.6K, ../../TY3PR01MB98890AB22AECA2F503EB7858F5742@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v7-0003-Fix-publication-does-not-exist-error.patch)
download | inline diff:
From fe1c57b974a2228b5ab2349b31de16d04db24aac Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:36:20 +0530
Subject: [PATCH v7 3/3] Fix publication does not exist error.
Fix publication does not exist error.
---
src/bin/pg_basebackup/pg_subscriber.c | 23 +++--------------------
1 file changed, 3 insertions(+), 20 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 3880d15ef9..355738c20c 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -679,6 +679,9 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ /* Also create a publication */
+ create_publication(conn, primary, perdb);
+
disconnect_database(conn);
}
@@ -1794,26 +1797,6 @@ main(int argc, char **argv)
*/
wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
- /*
- * Create a publication for each database. This step should be executed
- * after promoting the subscriber to avoid replicating unnecessary
- * objects.
- */
- for (i = 0; i < dbarr.ndbs; i++)
- {
- LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
-
- /* Connect to publisher. */
- conn = connect_database(primary.base_conninfo, perdb->dbname);
- if (conn == NULL)
- exit(1);
-
- /* Also create a publication */
- create_publication(conn, &primary, perdb);
-
- disconnect_database(conn);
- }
-
/*
* Create a subscription for each database.
*/
--
2.43.0
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-24 06:11 ` Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Shubham Khanna @ 2024-01-24 06:11 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Shlok Kyal <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
On Tue, Jan 23, 2024 at 7:41 AM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Dear hackers,
>
> > We fixed some of the comments posted in the thread. We have created it
> > as top-up patch 0002 and 0003.
>
> I found that the CFbot raised an ERROR. Also, it may not work well in case
> of production build.
> PSA the fixed patch set.
Segmentation fault was found after testing the given command(There is
an extra '/' between 'new_standby2' and '-P') '$ gdb --args
./pg_subscriber -D ../new_standby2 / -P "host=localhost
port=5432 dbname=postgres" -d postgres'
While executing the above command, I got the following error:
pg_subscriber: error: too many command-line arguments (first is "/")
pg_subscriber: hint: Try "pg_subscriber --help" for more information.
Program received signal SIGSEGV, Segmentation fault.
0x0000555555557e5b in cleanup_objects_atexit () at pg_subscriber.c:173
173 if (perdb->made_subscription)
(gdb) p perdb
$1 = (LogicalRepPerdbInfo *) 0x0
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
@ 2024-01-24 11:43 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-24 11:43 UTC (permalink / raw)
To: 'Shubham Khanna' <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
Dear Shubham,
> Segmentation fault was found after testing the given command(There is
> an extra '/' between 'new_standby2' and '-P') '$ gdb --args
> ./pg_subscriber -D ../new_standby2 / -P "host=localhost
> port=5432 dbname=postgres" -d postgres'
> While executing the above command, I got the following error:
> pg_subscriber: error: too many command-line arguments (first is "/")
> pg_subscriber: hint: Try "pg_subscriber --help" for more information.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000555555557e5b in cleanup_objects_atexit () at pg_subscriber.c:173
> 173 if (perdb->made_subscription)
> (gdb) p perdb
> $1 = (LogicalRepPerdbInfo *) 0x0
>
Good catch, I could reproduce the issue. This crash was occurred because the
cleanup function was called before initialization memory.
There are several ways to fix it, but I chose to move the callback registration
behind. The function does actual tasks only after database objects are created.
So 0004 registers the function just before doing them. The memory allocation has
been done at that time. If required, Assert() can be added in the callback.
Can you test it and confirm the issue was solved?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v8-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (67.0K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v8-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From 26bc1ee9371409e360588ac6aacafaf4fafb5e96 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v8 1/4] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 284 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1657 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2153 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..553185c35f
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,284 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>create a new logical replica from a standby server</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a standby server and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a standby
+ server.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a standby server. Stop the standby server if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a standby server at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..e998c29f9e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1657 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_subscriber/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static char *use_primary_slot_name(void);
+static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name plus a fallback application name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Return a palloc'd slot name if the replication is using one.
+ */
+static char *
+use_primary_slot_name(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+ char *slot_name;
+
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings WHERE name = 'primary_slot_name'");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain parameter information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ /*
+ * If primary_slot_name is an empty string, the current replication
+ * connection is not using a replication slot, bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "") == 0)
+ {
+ PQclear(res);
+ return NULL;
+ }
+
+ slot_name = pg_strdup(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots r INNER JOIN pg_stat_activity a ON (r.active_pid = a.pid) WHERE slot_name = '%s'", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ return NULL;
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ return slot_name;
+}
+
+static bool
+create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+{
+ int i;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ PGconn *conn;
+ PGresult *res;
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * Stop the subscriber if it is a standby server. Before executing the
+ * transformation steps, make sure the subscriber is not running because
+ * one of the steps is to modify some recovery parameters that require a
+ * restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Since the standby server is running, check if it is using an
+ * existing replication slot for WAL retention purposes. This
+ * replication slot has no use after the transformation, hence, it
+ * will be removed at the end of this process.
+ */
+ primary_slot_name = use_primary_slot_name();
+ if (primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+
+ pg_log_info("subscriber is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+
+ /*
+ * Create a replication slot for each database on the publisher.
+ */
+ if (!create_all_logical_replication_slots(dbinfo))
+ exit(1);
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a publication for each database. This step should be executed
+ * after promoting the subscriber to avoid replicating unnecessary
+ * objects.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+
+ /* Connect to publisher. */
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ create_publication(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * Remove log file generated by this tool, if it runs successfully.
+ * Otherwise, file is kept that may provide useful debugging information.
+ */
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.43.0
[application/octet-stream] v8-0002-Address-some-comments-proposed-on-hackers.patch (56.5K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v8-0002-Address-some-comments-proposed-on-hackers.patch)
download | inline diff:
From 327fa75f88f913b4731e73b066e1d30b9225ed44 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:42:34 +0530
Subject: [PATCH v8 2/4] Address some comments proposed on -hackers
The patch has following changes:
* Some comments reported on the thread
* Add a timeout option for the recovery option
* Reject if the target server is not a standby
* Reject when the --subscriber-conninfo specifies non-local server
* Add -u and -p options
* Check wal_level and max_replication_slot parameters
---
doc/src/sgml/ref/pg_subscriber.sgml | 21 +-
src/bin/pg_basebackup/pg_subscriber.c | 911 +++++++++++-------
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 9 +-
.../t/041_pg_subscriber_standby.pl | 8 +-
4 files changed, 601 insertions(+), 348 deletions(-)
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
index 553185c35f..eaabfc7053 100644
--- a/doc/src/sgml/ref/pg_subscriber.sgml
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -16,12 +16,18 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_subscriber</refname>
- <refpurpose>create a new logical replica from a standby server</refpurpose>
+ <refpurpose>Convert a standby replica to a logical replica</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_subscriber</command>
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><replaceable>datadir</replaceable></arg>
+ <arg choice="plain"><option>-P</option>
+ <replaceable>publisher-conninfo</replaceable></arg>
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><replaceable>subscriber-conninfo</replaceable></arg>
<arg rep="repeat"><replaceable>option</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
@@ -29,17 +35,18 @@ PostgreSQL documentation
<refsect1>
<title>Description</title>
<para>
- <application>pg_subscriber</application> takes the publisher and subscriber
- connection strings, a cluster directory from a standby server and a list of
- database names and it sets up a new logical replica using the physical
- recovery process.
+ pg_subscriber creates a new <link
+ linkend="logical-replication-subscription">subscriber</link> from a physical
+ standby server. This allows users to quickly set up logical replication
+ system.
</para>
<para>
- The <application>pg_subscriber</application> should be run at the target
+ The <application>pg_subscriber</application> has to be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
+ The target server should accept logical replication connection from
+ localhost.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index e998c29f9e..3880d15ef9 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1,12 +1,12 @@
/*-------------------------------------------------------------------------
*
* pg_subscriber.c
- * Create a new logical replica from a standby server
+ * Convert a standby replica to a logical replica
*
* Copyright (C) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/bin/pg_subscriber/pg_subscriber.c
+ * src/bin/pg_basebackup/pg_subscriber.c
*
*-------------------------------------------------------------------------
*/
@@ -32,81 +32,122 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
-static bool get_exec_path(const char *path);
+static char *get_base_conninfo(char *conninfo, char *dbname);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_control_for_standby(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static char *use_primary_slot_name(void);
-static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb);
+static bool create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void modify_sysid(const char *bindir, const char *datadir);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void wait_for_end_recovery(const char *base_conninfo,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ char *base_conninfo,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log);
+static char *construct_sub_conninfo(char *username, unsigned short subport);
#define USEC_PER_SEC 1000000
-#define WAIT_INTERVAL 1 /* 1 second */
+#define DEFAULT_WAIT 60
+#define WAITS_PER_SEC 10 /* should divide USEC_PER_SEC evenly */
+#define DEF_PGSPORT 50111
/* Options */
-static const char *progname;
-
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
-static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
+static int wait_seconds = DEFAULT_WAIT;
+static bool retain = false;
static bool dry_run = false;
static bool success = false;
+static const char *progname;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
+enum PGSWaitPMResult
+{
+ PGS_POSTMASTER_READY,
+ PGS_POSTMASTER_STANDBY,
+ PGS_POSTMASTER_STILL_STARTING,
+ PGS_POSTMASTER_FAILED
+};
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
-enum WaitPMResult
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
{
- POSTMASTER_READY,
- POSTMASTER_STANDBY,
- POSTMASTER_STILL_STARTING,
- POSTMASTER_FAILED
-};
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
/*
@@ -125,41 +166,39 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
disconnect_database(conn);
}
}
}
-
- if (made_transient_replslot)
- {
- conn = connect_database(dbinfo[0].pubconninfo);
- if (conn != NULL)
- {
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
- disconnect_database(conn);
- }
- }
}
static void
@@ -184,17 +223,16 @@ usage(void)
/*
* Validate a connection string. Returns a base connection string that is a
- * connection string without a database name plus a fallback application name.
- * Since we might process multiple databases, each database name will be
- * appended to this base connection string to provide a final connection string.
- * If the second argument (dbname) is not null, returns dbname if the provided
- * connection string contains it. If option --database is not provided, uses
- * dbname as the only database to setup the logical replica.
- * It is the caller's responsibility to free the returned connection string and
- * dbname.
+ * connection string without a database name. Since we might process multiple
+ * databases, each database name will be appended to this base connection
+ * string to provide a final connection string. If the second argument (dbname)
+ * is not null, returns dbname if the provided connection string contains it.
+ * If option --database is not provided, uses dbname as the only database to
+ * setup the logical replica. It is the caller's responsibility to free the
+ * returned connection string and dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -203,7 +241,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on publisher");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -231,10 +269,6 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
}
- if (i > 0)
- appendPQExpBufferChar(buf, ' ');
- appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
-
ret = pg_strdup(buf->data);
destroyPQExpBuffer(buf);
@@ -244,15 +278,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -277,30 +312,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -364,49 +381,36 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
SimpleStringListCell *cell;
int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ *perdb = (LogicalRepPerdbInfo *) pg_malloc0(sizeof(LogicalRepPerdbInfo) *
+ ndbs);
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -424,6 +428,9 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
+
return conn;
}
@@ -436,19 +443,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -471,43 +477,39 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_control_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -516,7 +518,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -551,7 +553,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -571,14 +573,15 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* Return a palloc'd slot name if the replication is using one.
*/
static char *
-use_primary_slot_name(void)
+use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb)
{
PGconn *conn;
PGresult *res;
PQExpBuffer str = createPQExpBuffer();
char *slot_name;
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -604,7 +607,7 @@ use_primary_slot_name(void)
disconnect_database(conn);
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -634,17 +637,19 @@ use_primary_slot_name(void)
}
static bool
-create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr)
{
int i;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr->ndbs; i++)
{
PGconn *conn;
PGresult *res;
char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -664,27 +669,14 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
disconnect_database(conn);
@@ -701,30 +693,36 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (!temporary)
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name, NAMEDATALEN);
+ else
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+
+ if(temporary)
+ appendPQExpBufferStr(str, " TEMPORARY");
+
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
pg_log_debug("command is: %s", str->data);
@@ -734,17 +732,14 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
- else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -758,14 +753,15 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -775,7 +771,7 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -825,19 +821,22 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* Returns after the server finishes the recovery process.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(const char *base_conninfo, const char *dbname)
{
PGconn *conn;
PGresult *res;
- int status = POSTMASTER_STILL_STARTING;
+ int status = PGS_POSTMASTER_STILL_STARTING;
+ int cnt;
+ int rc;
+ char *pg_ctl_cmd;
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(base_conninfo, dbname);
if (conn == NULL)
exit(1);
- for (;;)
+ for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -865,17 +864,32 @@ wait_for_end_recovery(const char *conninfo)
*/
if (!in_recovery || dry_run)
{
- status = POSTMASTER_READY;
+ status = PGS_POSTMASTER_READY;
break;
}
/* Keep waiting. */
- pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
disconnect_database(conn);
- if (status == POSTMASTER_STILL_STARTING)
+ /*
+ * If timeout is reached exit the pg_subscriber and stop the standby node.
+ */
+ if (cnt >= wait_seconds * WAITS_PER_SEC)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ if (status == PGS_POSTMASTER_STILL_STARTING)
{
pg_log_error("server did not end recovery");
exit(1);
@@ -888,17 +902,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -918,7 +936,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -931,7 +949,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -942,9 +960,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -954,14 +972,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -973,16 +991,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
@@ -990,7 +1011,7 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1011,19 +1032,27 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname, concat_conninfo_dbname(base_conninfo, perdb->dbname), pubname);
pg_log_debug("command is: %s", str->data);
@@ -1033,14 +1062,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1052,16 +1081,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1069,7 +1101,7 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1088,18 +1120,21 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1140,7 +1175,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1154,7 +1189,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1173,16 +1208,20 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1191,7 +1230,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1203,6 +1242,61 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+static void
+start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ int rc;
+ char *pg_ctl_cmd;
+
+ if (server_start_log[0] == '\0')
+ {
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
+ standby->bindir,
+ standby->pgdata, subport, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static char *
+construct_sub_conninfo(char *username, unsigned short subport)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ if (username)
+ appendPQExpBuffer(buf, "user=%s ", username);
+
+ appendPQExpBuffer(buf, "port=%d fallback_application_name=%s",
+ subport, progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
int
main(int argc, char **argv)
{
@@ -1214,6 +1308,10 @@ main(int argc, char **argv)
{"publisher-conninfo", required_argument, NULL, 'P'},
{"subscriber-conninfo", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
+ {"timeout", required_argument, NULL, 't'},
+ {"username", required_argument, NULL, 'u'},
+ {"port", required_argument, NULL, 'p'},
+ {"retain", no_argument, NULL, 'r'},
{"dry-run", no_argument, NULL, 'n'},
{"verbose", no_argument, NULL, 'v'},
{NULL, 0, NULL, 0}
@@ -1225,20 +1323,15 @@ main(int argc, char **argv)
char *pg_ctl_cmd;
- char *base_dir;
- char *server_start_log;
-
- char timebuf[128];
- struct timeval time;
- time_t tt;
+ char base_dir[MAXPGPATH];
+ char server_start_log[MAXPGPATH] = {0};
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
+ unsigned short subport = DEF_PGSPORT;
+ char *username = NULL;
+
struct stat statbuf;
PGconn *conn;
@@ -1250,6 +1343,13 @@ main(int argc, char **argv)
int i;
+ PGresult *res;
+
+ char *wal_level;
+ int max_replication_slots;
+ int nslots_old;
+ int nslots_new;
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
progname = get_progname(argv[0]);
@@ -1286,28 +1386,40 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:t:u:p:rnv",
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
+ canonicalize_path(standby.pgdata);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
break;
- case 'S':
- sub_conninfo_str = pg_strdup(optarg);
- break;
case 'd':
/* Ignore duplicated database names. */
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
+ case 't':
+ wait_seconds = atoi(optarg);
+ break;
+ case 'u':
+ pfree(username);
+ username = pg_strdup(optarg);
+ break;
+ case 'p':
+ if ((subport = atoi(optarg)) <= 0)
+ pg_fatal("invalid old port number");
+ break;
+ case 'r':
+ retain = true;
+ break;
case 'n':
dry_run = true;
break;
@@ -1335,7 +1447,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1358,21 +1470,14 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
- if (sub_conninfo_str == NULL)
- {
- pg_log_error("no subscriber connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo);
+ if (primary.base_conninfo == NULL)
exit(1);
+ standby.base_conninfo = construct_sub_conninfo(username, subport);
+
if (database_names.head == NULL)
{
pg_log_info("no database was specified");
@@ -1385,7 +1490,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1399,25 +1504,25 @@ main(int argc, char **argv)
}
/*
- * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ * Get the absolute path of binaries on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_control_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1426,8 +1531,8 @@ main(int argc, char **argv)
/*
* Create the output directory to store any data generated by this tool.
*/
- base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s",
+ standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1441,7 +1546,153 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid",
+ standby.pgdata);
+
+ /* Start the standby server anyway */
+ start_standby_server(&standby, subport, server_start_log);
+
+ /*
+ * Check wal_level in publisher and the max_replication_slots of publisher
+ */
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name IN ('wal_level', 'max_replication_slots') "
+ "ORDER BY name DESC;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters on publisher");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 2)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ wal_level = PQgetvalue(res, 0, 0);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("wal_level must be \"logical\", but is set to \"%s\"", wal_level);
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 1, 0));
+ nslots_new = nslots_old + dbarr.ndbs + 1;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ conn = connect_database(standby.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Check the max_replication_slots in subscriber
+ */
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots on subscriber");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on subscriber");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'max_replication_slots';");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 0, 0));
+ nslots_new = nslots_old + dbarr.ndbs;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+
+ /*
+ * Exit the pg_subscriber if the node is not a standby server.
+ */
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ /* Check if the server is in recovery */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") != 0)
+ {
+ pg_log_error("pg_subscriber is supported only on standby server");
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* Stop the subscriber if it is a standby server. Before executing the
@@ -1457,14 +1708,18 @@ main(int argc, char **argv)
* replication slot has no use after the transformation, hence, it
* will be removed at the end of this process.
*/
- primary_slot_name = use_primary_slot_name();
- if (primary_slot_name != NULL)
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ standby.primary_slot_name = use_primary_slot_name(&primary,
+ &standby,
+ &dbarr.perdb[0]);
+ if (standby.primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
pg_log_info("subscriber is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1472,7 +1727,7 @@ main(int argc, char **argv)
/*
* Create a replication slot for each database on the publisher.
*/
- if (!create_all_logical_replication_slots(dbinfo))
+ if (!create_all_logical_replication_slots(&primary, &dbarr))
exit(1);
/*
@@ -1492,11 +1747,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true,
+ &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1522,7 +1777,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1532,54 +1787,29 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby_server(&standby, subport, server_start_log);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
/*
* Create a publication for each database. This step should be executed
* after promoting the subscriber to avoid replicating unnecessary
* objects.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- char pubname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
/* Connect to publisher. */
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
- create_publication(conn, &dbinfo[i]);
+ /* Also create a publication */
+ create_publication(conn, &primary, perdb);
disconnect_database(conn);
}
@@ -1587,20 +1817,25 @@ main(int argc, char **argv)
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
+
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, primary.base_conninfo, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
+
+ drop_publication(conn, perdb);
disconnect_database(conn);
}
@@ -1613,19 +1848,21 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
- pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
disconnect_database(conn);
}
@@ -1634,20 +1871,22 @@ main(int argc, char **argv)
*/
pg_log_info("stopping the subscriber");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* Remove log file generated by this tool, if it runs successfully.
* Otherwise, file is kept that may provide useful debugging information.
*/
- unlink(server_start_log);
+ if (!retain)
+ unlink(server_start_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
index 4ebff76b2d..9915b8cb3c 100644
--- a/src/bin/pg_basebackup/t/040_pg_subscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -37,8 +37,13 @@ command_fails(
'--verbose',
'--pgdata', $datadir,
'--publisher-conninfo', 'dbname=postgres',
- '--subscriber-conninfo', 'dbname=postgres'
],
'no database name specified');
-
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ ],
+ 'subscriber connection string specnfied non-local server');
done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
index fbcd0fc82b..4e26607611 100644
--- a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -51,25 +51,27 @@ $node_s->start;
$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
$node_p->wait_for_replay_catchup($node_s);
+$node_f->stop;
+
# Run pg_subscriber on about-to-fail node F
command_fails(
[
'pg_subscriber', '--verbose',
'--pgdata', $node_f->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
'subscriber data directory is not a copy of the source database cluster');
+$node_s->stop;
+
# dry run mode on node S
command_ok(
[
'pg_subscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
@@ -82,6 +84,7 @@ $node_s->start;
# Check if node S is still a standby
is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
+$node_s->stop;
# Run pg_subscriber on node S
command_ok(
@@ -89,7 +92,6 @@ command_ok(
'pg_subscriber', '--verbose',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
--
2.43.0
[application/octet-stream] v8-0003-Fix-publication-does-not-exist-error.patch (1.6K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v8-0003-Fix-publication-does-not-exist-error.patch)
download | inline diff:
From 9f538ed90a02fb13404ff0df4884d1e809a13f1c Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:36:20 +0530
Subject: [PATCH v8 3/4] Fix publication does not exist error.
Fix publication does not exist error.
---
src/bin/pg_basebackup/pg_subscriber.c | 23 +++--------------------
1 file changed, 3 insertions(+), 20 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 3880d15ef9..355738c20c 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -679,6 +679,9 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ /* Also create a publication */
+ create_publication(conn, primary, perdb);
+
disconnect_database(conn);
}
@@ -1794,26 +1797,6 @@ main(int argc, char **argv)
*/
wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
- /*
- * Create a publication for each database. This step should be executed
- * after promoting the subscriber to avoid replicating unnecessary
- * objects.
- */
- for (i = 0; i < dbarr.ndbs; i++)
- {
- LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
-
- /* Connect to publisher. */
- conn = connect_database(primary.base_conninfo, perdb->dbname);
- if (conn == NULL)
- exit(1);
-
- /* Also create a publication */
- create_publication(conn, &primary, perdb);
-
- disconnect_database(conn);
- }
-
/*
* Create a subscription for each database.
*/
--
2.43.0
[application/octet-stream] v8-0004-Move-a-registration-of-atexit-callback-to-behind.patch (1.1K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/5-v8-0004-Move-a-registration-of-atexit-callback-to-behind.patch)
download | inline diff:
From eb912015aa8b5d8481b852d91e3d146fc1a31703 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 24 Jan 2024 07:18:47 +0000
Subject: [PATCH v8 4/4] Move a registration of atexit() callback to behind
---
src/bin/pg_basebackup/pg_subscriber.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 355738c20c..17a6b552af 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1373,8 +1373,6 @@ main(int argc, char **argv)
}
}
- atexit(cleanup_objects_atexit);
-
/*
* Don't allow it to be run as root. It uses pg_ctl which does not allow
* it either.
@@ -1727,6 +1725,12 @@ main(int argc, char **argv)
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
+ /*
+ * Subsequent operations define some database objects on both primary and
+ * standby. The callback is useful to clean up them in case of failure.
+ */
+ atexit(cleanup_objects_atexit);
+
/*
* Create a replication slot for each database on the publisher.
*/
--
2.43.0
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-25 09:05 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-26 00:28 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-25 09:05 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>; 'Shubham Khanna' <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>
Dear hackers,
Here are comments for v8 patch set. I may revise them by myself,
but I want to post here to share all of them.
01.
```
/* Options */
static char *pub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
static int wait_seconds = DEFAULT_WAIT;
static bool retain = false;
static bool dry_run = false;
```
Just to confirm - is there a policy why we store the specified options? If you
want to store as global ones, username and port should follow (my fault...).
Or, should we have a structure to store them?
02.
```
{"subscriber-conninfo", required_argument, NULL, 'S'},
```
This is my fault, but "--subscriber-conninfo" is still remained. It should be
removed if it is not really needed.
03.
```
{"username", required_argument, NULL, 'u'},
```
Should we accept 'U' instead of 'u'?
04.
```
{"dry-run", no_argument, NULL, 'n'},
```
I'm not sure why the dry_run mode exists. In terms pg_resetwal, it shows the
which value would be changed based on the input. As for the pg_upgrade, it checks
whether the node can be upgraded for now. I think, we should have the checking
feature, so it should be renamed to --check. Also, the process should exit earlier
at that time.
05.
I felt we should accept some settings from enviroment variables, like pg_upgrade.
Currently, below items should be acceted.
- data directory
- username
- port
- timeout
06.
```
pg_logging_set_level(PG_LOG_WARNING);
```
If the default log level is warning, there are no ways to output debug logs.
(-v option only raises one, so INFO would be output)
I think it should be PG_LOG_INFO.
07.
Can we combine verifications into two functions, e.g., check_primary() and check_standby/check_subscriber()?
08.
Not sure, but if we want to record outputs by pg_subscriber, the sub-directory
should be created. The name should contain the timestamp.
09.
Not sure, but should we check max_slot_wal_keep_size of primary server? It can
avoid to fail starting of logical replicaiton.
10.
```
nslots_new = nslots_old + dbarr.ndbs;
if (nslots_new > max_replication_slots)
{
pg_log_error("max_replication_slots (%d) must be greater than or equal to "
"the number of replication slots required (%d)", max_replication_slots, nslots_new);
exit(1);
}
```
I think standby server must not have replication slots. Because subsequent
pg_resetwal command discards all the WAL file, so WAL records pointed by them
are removed. Currently pg_resetwal does not raise ERROR at that time.
11.
```
/*
* Stop the subscriber if it is a standby server. Before executing the
* transformation steps, make sure the subscriber is not running because
* one of the steps is to modify some recovery parameters that require a
* restart.
*/
if (stat(pidfile, &statbuf) == 0)
```
I kept just in case, but I'm not sure it is still needed. How do you think?
Removing it can reduce an inclusion of pidfile.h.
12.
```
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
```
There are two places to stop the instance. Can you divide it into a function?
13.
```
* A temporary replication slot is not used here to avoid keeping a
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
consistent_lsn = create_logical_replication_slot(conn, true,
&dbarr.perdb[0]);
```
I didn't notice the comment, but still not sure the reason. Why we must reserve
the slot until pg_subscriber finishes? IIUC, the slot would be never used, it
is created only for getting a consistent_lsn. So we do not have to keep.
Also, just before, logical replication slots for each databases are created, so
WAL records are surely reserved.
14.
```
pg_log_info("starting the subscriber");
start_standby_server(&standby, subport, server_start_log);
```
This info should be in the function.
15.
```
/*
* Create a subscription for each database.
*/
for (i = 0; i < dbarr.ndbs; i++)
```
This can be divided into a function, like create_all_subscriptions().
16.
My fault: usage() must be updated.
17. use_primary_slot_name
```
if (PQntuples(res) != 1)
{
pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
PQntuples(res), 1);
return NULL;
}
```
Error message should be changed. I think this error means the standby has wrong primary_slot_name, right?
18. misc
Sometimes the pid of pg_subscriber is referred. It can be stored as global variable.
19.
C99-style has been allowed, so loop variables like "i" can be declared in the for-statement, like
```
for (int i = 0; i < MAX; i++)
```
20.
Some comments, docs, and outputs must be fixed when the name is changed.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-26 00:28 ` Euler Taveira <[email protected]>
2024-01-26 05:51 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Euler Taveira @ 2024-01-26 00:28 UTC (permalink / raw)
To: [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; 'Shubham Khanna' <[email protected]>
On Thu, Jan 25, 2024, at 6:05 AM, Hayato Kuroda (Fujitsu) wrote:
> 01.
> ```
> /* Options */
> static char *pub_conninfo_str = NULL;
> static SimpleStringList database_names = {NULL, NULL};
> static int wait_seconds = DEFAULT_WAIT;
> static bool retain = false;
> static bool dry_run = false;
> ```
>
> Just to confirm - is there a policy why we store the specified options? If you
> want to store as global ones, username and port should follow (my fault...).
> Or, should we have a structure to store them?
It is a matter of style I would say. Check other client applications. Some of
them also use global variable. There are others that group options into a
struct. I would say that since it has a short lifetime, I don't think the
current style is harmful.
> 04.
> ```
> {"dry-run", no_argument, NULL, 'n'},
> ```
>
> I'm not sure why the dry_run mode exists. In terms pg_resetwal, it shows the
> which value would be changed based on the input. As for the pg_upgrade, it checks
> whether the node can be upgraded for now. I think, we should have the checking
> feature, so it should be renamed to --check. Also, the process should exit earlier
> at that time.
It is extremely useful because (a) you have a physical replication setup and
don't know if it is prepared for logical replication, (b) check GUCs (is
max_wal_senders sufficient for this pg_subscriber command? Or is
max_replication_slots sufficient to setup the logical replication even though I
already have some used replication slots?), (c) connectivity and (d)
credentials.
> 05.
> I felt we should accept some settings from enviroment variables, like pg_upgrade.
> Currently, below items should be acceted.
>
> - data directory
> - username
> - port
> - timeout
Maybe PGDATA.
> 06.
> ```
> pg_logging_set_level(PG_LOG_WARNING);
> ```
>
> If the default log level is warning, there are no ways to output debug logs.
> (-v option only raises one, so INFO would be output)
> I think it should be PG_LOG_INFO.
You need to specify multiple -v options.
> 07.
> Can we combine verifications into two functions, e.g., check_primary() and check_standby/check_subscriber()?
I think v9 does it.
> 08.
> Not sure, but if we want to record outputs by pg_subscriber, the sub-directory
> should be created. The name should contain the timestamp.
The log file already contains the timestamp. Why?
> 09.
> Not sure, but should we check max_slot_wal_keep_size of primary server? It can
> avoid to fail starting of logical replicaiton.
A broken physical replication *before* running this tool is its responsibility?
Hmm. We might add another check that can be noticed during dry run mode.
> 10.
> ```
> nslots_new = nslots_old + dbarr.ndbs;
>
> if (nslots_new > max_replication_slots)
> {
> pg_log_error("max_replication_slots (%d) must be greater than or equal to "
> "the number of replication slots required (%d)", max_replication_slots, nslots_new);
> exit(1);
> }
> ```
>
> I think standby server must not have replication slots. Because subsequent
> pg_resetwal command discards all the WAL file, so WAL records pointed by them
> are removed. Currently pg_resetwal does not raise ERROR at that time.
Again, dry run mode might provide a message for it.
> 11.
> ```
> /*
> * Stop the subscriber if it is a standby server. Before executing the
> * transformation steps, make sure the subscriber is not running because
> * one of the steps is to modify some recovery parameters that require a
> * restart.
> */
> if (stat(pidfile, &statbuf) == 0)
> ```
>
> I kept just in case, but I'm not sure it is still needed. How do you think?
> Removing it can reduce an inclusion of pidfile.h.
Are you suggesting another way to check if the standby is up and running?
> 12.
> ```
> pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
> standby.bindir, standby.pgdata);
> rc = system(pg_ctl_cmd);
> pg_ctl_status(pg_ctl_cmd, rc, 0);
> ```
>
>
> There are two places to stop the instance. Can you divide it into a function?
Yes.
> 13.
> ```
> * A temporary replication slot is not used here to avoid keeping a
> * replication connection open (depending when base backup was taken, the
> * connection should be open for a few hours).
> */
> conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
> if (conn == NULL)
> exit(1);
> consistent_lsn = create_logical_replication_slot(conn, true,
> &dbarr.perdb[0]);
> ```
>
> I didn't notice the comment, but still not sure the reason. Why we must reserve
> the slot until pg_subscriber finishes? IIUC, the slot would be never used, it
> is created only for getting a consistent_lsn. So we do not have to keep.
> Also, just before, logical replication slots for each databases are created, so
> WAL records are surely reserved.
This comment needs to be updated. It was written at the time I was pursuing
base backup support too. It doesn't matter if you remove this transient
replication slot earlier because all of the replication slots created to the
subscriptions were created *before* the one for the consistent LSN. Hence, no
additional WAL retention due to this transient replication slot.
> 14.
>
> ```
> pg_log_info("starting the subscriber");
> start_standby_server(&standby, subport, server_start_log);
> ```
>
> This info should be in the function.
Ok.
> 15.
> ```
> /*
> * Create a subscription for each database.
> */
> for (i = 0; i < dbarr.ndbs; i++)
> ```
>
> This can be divided into a function, like create_all_subscriptions().
Ok.
> 16.
> My fault: usage() must be updated.
>
> 17. use_primary_slot_name
> ```
> if (PQntuples(res) != 1)
> {
> pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
> PQntuples(res), 1);
> return NULL;
> }
> ```
>
> Error message should be changed. I think this error means the standby has wrong primary_slot_name, right?
I refactored this code a bit but the message is the same. It detects 2 cases:
(a) you set primary_slot_name but you don't have a replication slot with the
same name and (b) a cannot-happen bug that provides > 1 rows. It is a broken
setup so maybe a hint saying so is enough.
> 18. misc
> Sometimes the pid of pg_subscriber is referred. It can be stored as global variable.
I prefer to keep getpid() call.
> 19.
> C99-style has been allowed, so loop variables like "i" can be declared in the for-statement, like
>
> ```
> for (int i = 0; i < MAX; i++)
> ```
v9 does it.
> 20.
> Some comments, docs, and outputs must be fixed when the name is changed.
Next patch.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-26 00:28 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-26 05:51 ` Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-26 05:51 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; 'Shubham Khanna' <[email protected]>
Dear Euler,
Thanks for updating the patch! Before reading yours, I wanted to reply some of comments.
>
I'm still thinking about replacing --subscriber-conninfo with separate items
(username, port, password?, host = socket dir). Maybe it is an overengineering.
The user can always prepare the environment to avoid unwanted and/or external
connections.
>
For me, required amount of fixes are not so different from current one. How about
others?
>
It is extremely useful because (a) you have a physical replication setup and
don't know if it is prepared for logical replication, (b) check GUCs (is
max_wal_senders sufficient for this pg_subscriber command? Or is
max_replication_slots sufficient to setup the logical replication even though I
already have some used replication slots?), (c) connectivity and (d)
credentials.
>
Yeah, it is useful for verification purpose, so let's keep this option.
But I still think the naming should be "--check". Also, there are many
`if (!dry_run)` but most of them can be removed if the process exits earlier.
Thought?
>
> 05.
> I felt we should accept some settings from enviroment variables, like pg_upgrade.
> Currently, below items should be acceted.
>
> - data directory
> - username
> - port
> - timeout
Maybe PGDATA.
>
Sorry, I cannot follow this. Did you mean that the target data directory should
be able to be specified by PGDATA? OF so, +1.
>
> 06.
> ```
> pg_logging_set_level(PG_LOG_WARNING);
> ```
>
> If the default log level is warning, there are no ways to output debug logs.
> (-v option only raises one, so INFO would be output)
> I think it should be PG_LOG_INFO.
You need to specify multiple -v options.
>
Hmm. I felt the specification was bit strange...but at least it must be
described on the documentation. pg_dump.sgml has similar lines.
>
> 08.
> Not sure, but if we want to record outputs by pg_subscriber, the sub-directory
> should be created. The name should contain the timestamp.
The log file already contains the timestamp. Why?
>
This comment assumed outputs by pg_subscriber were also recorded to a file.
In this case and if the file also has the same timestamp, I think they can be
gathered in the same place. No need if outputs are not recorded.
>
> 09.
> Not sure, but should we check max_slot_wal_keep_size of primary server? It can
> avoid to fail starting of logical replicaiton.
A broken physical replication *before* running this tool is its responsibility?
Hmm. We might add another check that can be noticed during dry run mode.
>
I thought that we should not generate any broken objects, but indeed, not sure
it is our scope. How do other think?
>
> 11.
> ```
> /*
> * Stop the subscriber if it is a standby server. Before executing the
> * transformation steps, make sure the subscriber is not running because
> * one of the steps is to modify some recovery parameters that require a
> * restart.
> */
> if (stat(pidfile, &statbuf) == 0)
> ```
>
> I kept just in case, but I'm not sure it is still needed. How do you think?
> Removing it can reduce an inclusion of pidfile.h.
Are you suggesting another way to check if the standby is up and running?
>
Running `pg_ctl stop` itself can detect whether the process has been still alive.
It would exit with 1 when the process is not there.
>
> I didn't notice the comment, but still not sure the reason. Why we must reserve
> the slot until pg_subscriber finishes? IIUC, the slot would be never used, it
> is created only for getting a consistent_lsn. So we do not have to keep.
> Also, just before, logical replication slots for each databases are created, so
> WAL records are surely reserved.
>
I want to confirm the conclusion - will you remove the creation of a transient slot?
Also, not tested, I'm now considering that we can reuse the primary_conninfo value.
We are assuming that the target server is standby and the current upstream one will
convert to publisher. In this case, the connection string is already specified as
primary_conninfo so --publisher-conninfo may not be needed. The parameter does
not contain database name, so --databases is still needed. I imagine like:
1. Parse options
2. Turn on standby
3. Verify the standby
4. Turn off standby
5. Get primary_conninfo from standby
6. Connect to primary (concat of primary_conninfo and an option is used)
7. Verify the primary
...
How do you think?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
@ 2024-01-24 01:29 ` Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
1 sibling, 1 reply; 87+ messages in thread
From: Euler Taveira @ 2024-01-24 01:29 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Mon, Jan 22, 2024, at 6:30 AM, Shlok Kyal wrote:
> We fixed some of the comments posted in the thread. We have created it
> as top-up patch 0002 and 0003.
Cool.
> 0002 patch contains the following changes:
> * Add a timeout option for the recovery option, per [1]. The code was
> basically ported from pg_ctl.c.
> * Reject if the target server is not a standby, per [2]
> * Raise FATAL error if --subscriber-conninfo specifies non-local server, per [3]
> (not sure it is really needed, so feel free reject the part.)
> * Add check for max_replication_slots and wal_level; as per [4]
> * Add -u and -p options; as per [5]
> * Addressed comment except 5 and 8 in [6] and comment in [7]
My suggestion is that you create separate patches for each change. It helps
with review and alternative proposals. Some of these items conflict with what I
have in my local branch and removing one of them is time consuming. For this
one, I did the job but let's avoid rework.
> 0003 patch contains fix for bug reported in [8].
LGTM. As I said in the other email, I included it.
I'll post a new one soon.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-25 23:34 ` Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Euler Taveira @ 2024-01-25 23:34 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Tue, Jan 23, 2024, at 10:29 PM, Euler Taveira wrote:
> I'll post a new one soon.
I'm attaching another patch that fixes some of the issues pointed out by
Hayato, Shlok, and Junwang.
* publication doesn't exist. The analysis [1] was done by Hayato but I didn't
use the proposed patch. Instead I refactored the code a bit [2] and call it
from a new function (setup_publisher) that is called before the promotion.
* fix wrong path name in the initial comment [3]
* change terminology: logical replica -> physical replica [3]
* primary / standby is ready for logical replication? setup_publisher() and
setup_subscriber() check if required GUCs are set accordingly. For primary,
it checks wal_level = logical, max_replication_slots has remain replication
slots for the proposed setup and also max_wal_senders available. For standby,
it checks max_replication_slots for replication origin and also remain number
of background workers to start the subscriber.
* retain option: I extracted this one from Hayato's patch [4]
* target server must be a standby. It seems we agree that this restriction
simplifies the code a bit but can be relaxed in the future (if/when base
backup support is added.)
* recovery timeout option: I decided to include it but I think the use case is
too narrow. It helps in broken setups. However, it can be an issue in some
scenarios like time-delayed replica, large replication lag, slow hardware,
slow network. I didn't use the proposed patch [5]. Instead, I came up with a
simple one that defaults to forever. The proposed one defaults to 60 seconds
but I'm afraid that due to one of the scenarios I said in a previous
sentence, we cancel a legitimate case. Maybe we should add a message during
dry run saying that due to a replication lag, it will take longer to run.
* refactor primary_slot_name code. With the new setup_publisher and
setup_subscriber functions, I splitted the function that detects the
primary_slot_name use into 2 pieces just to avoid extra connections to have
the job done.
* remove fallback_application_name as suggested by Hayato [5] because logical
replication already includes one.
I'm still thinking about replacing --subscriber-conninfo with separate items
(username, port, password?, host = socket dir). Maybe it is an overengineering.
The user can always prepare the environment to avoid unwanted and/or external
connections.
I didn't change the name from pg_subscriber to pg_createsubscriber yet but if I
didn't hear objections about it, I'll do it in the next patch.
[1] https://www.postgresql.org/message-id/TY3PR01MB9889C5D55206DDD978627D07F5752%40TY3PR01MB9889.jpnprd0...
[2] https://www.postgresql.org/message-id/73ab86ca-3fd5-49b3-9c80-73d1525202f1%40app.fastmail.com
[3] https://www.postgresql.org/message-id/TY3PR01MB9889678E47B918F4D83A6FD8F57B2%40TY3PR01MB9889.jpnprd0...
[4] https://www.postgresql.org/message-id/TY3PR01MB9889678E47B918F4D83A6FD8F57B2%40TY3PR01MB9889.jpnprd0...
[5] https://www.postgresql.org/message-id/TY3PR01MB9889593399165B9A04106741F5662%40TY3PR01MB9889.jpnprd0...
--
Euler Taveira
EDB https://www.enterprisedb.com/
Attachments:
[text/x-patch] v9-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (73.2K, ../../[email protected]/3-v9-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From d9ef01a806c3d8697faa444283f19c2deaa58850 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v9] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 305 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1805 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2322 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..99d4fcee49
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,305 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..cb97dbda5e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1805 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Is the source server ready for logical replication? If so, create the
+ * publications and replication slots in preparation for logical replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the target server ready for logical replication?
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!setup_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication and
+ * create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.30.2
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-26 07:55 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-26 07:55 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; 'Euler Taveira' <[email protected]>; Shlok Kyal <[email protected]>
Dear Euler,
Again, thanks for updating the patch! There are my random comments for v9.
01.
I cannot find your replies for my comments#7 [1] but you reverted related changes.
I'm not sure you are still considering it or you decided not to include changes.
Can you clarify your opinion?
(It is needed because changes are huge so it quite affects other developments...)
02.
```
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
```
But source codes required `--recovery-timeout`. Please update either of them,
03.
```
+ * Create a new logical replica from a standby server
```
Junwang pointed out to change here but the change was reverted [2]
Can you clarify your opinion as well?
04.
```
+/*
+ * Is the source server ready for logical replication? If so, create the
+ * publications and replication slots in preparation for logical replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
```
But this function verifies the source server. I felt they should be in the
different function.
05.
```
+/*
+ * Is the target server ready for logical replication?
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo)
````
Actually, this function does not set up subscriber. It just verifies whether the
target can become a subscriber, right? If should be renamed.
06.
```
+ atexit(cleanup_objects_atexit);
```
The registration of the cleanup function is too early. This sometimes triggers
a core-dump. E.g.,
```
$ pg_subscriber --publisher-conninfo --subscriber-conninfo 'user=postgres port=5432' --verbose --database 'postgres' --pgdata data_N2/
pg_subscriber: error: too many command-line arguments (first is "user=postgres port=5432")
pg_subscriber: hint: Try "pg_subscriber --help" for more information.
Segmentation fault (core dumped)
$ gdb ...
(gdb) bt
#0 cleanup_objects_atexit () at pg_subscriber.c:131
#1 0x00007fb982cffce9 in __run_exit_handlers () from /lib64/libc.so.6
#2 0x00007fb982cffd37 in exit () from /lib64/libc.so.6
#3 0x00000000004054e6 in main (argc=9, argv=0x7ffc59074158) at pg_subscriber.c:1500
(gdb) f 3
#3 0x00000000004054e6 in main (argc=9, argv=0x7ffc59074158) at pg_subscriber.c:1500
1500 exit(1);
(gdb) list
1495 if (optind < argc)
1496 {
1497 pg_log_error("too many command-line arguments (first is \"%s\")",
1498 argv[optind]);
1499 pg_log_error_hint("Try \"%s --help\" for more information.", progname);
1500 exit(1);
1501 }
1502
1503 /*
1504 * Required arguments
```
I still think it should be done just before the creation of objects [3].
07.
Missing a removal of publications on the standby.
08.
Missing registration of LogicalRepInfo in the typedefs.list.
09
```
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
```
Can you reply my comment#2 [4]? I think mandatory options should be written.
10.
Just to confirm - will you implement start_standby/stop_standby functions in next version?
[1]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[2]: https://www.postgresql.org/message-id/CAEG8a3%2BwL_2R8n12BmRz7yBP3EBNdHDhmdgxQFA9vS%2BzPR%2B3Kw%40ma...
[3]: https://www.postgresql.org/message-id/TY3PR01MB9889678E47B918F4D83A6FD8F57B2%40TY3PR01MB9889.jpnprd0...
[4]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-29 01:10 ` Euler Taveira <[email protected]>
2024-01-29 09:57 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 2 replies; 87+ messages in thread
From: Euler Taveira @ 2024-01-29 01:10 UTC (permalink / raw)
To: [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Fri, Jan 26, 2024, at 4:55 AM, Hayato Kuroda (Fujitsu) wrote:
> Again, thanks for updating the patch! There are my random comments for v9.
Thanks for checking v9. I already incorporated some of the points below into
the next patch. Give me a couple of hours to include some important points.
> 01.
> I cannot find your replies for my comments#7 [1] but you reverted related changes.
> I'm not sure you are still considering it or you decided not to include changes.
> Can you clarify your opinion?
> (It is needed because changes are huge so it quite affects other developments...)
It is still on my list. As I said in a previous email I'm having a hard time
reviewing pieces from your 0002 patch because you include a bunch of things
into one patch.
> 02.
> ```
> + <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
> + <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
> ```
>
> But source codes required `--recovery-timeout`. Please update either of them,
Oops. Fixed. My preference is --recovery-timeout because someone can decide to
include a --timeout option for this tool.
> 03.
> ```
> + * Create a new logical replica from a standby server
> ```
>
> Junwang pointed out to change here but the change was reverted [2]
> Can you clarify your opinion as well?
I'll review the documentation once I fix the code. Since the tool includes the
*create* verb into its name, it seems strange use another verb (convert) in the
description. Maybe we should just remove the word *new* and a long description
explains the action is to turn the standby (physical replica) into a logical
replica.
> 04.
> ```
> +/*
> + * Is the source server ready for logical replication? If so, create the
> + * publications and replication slots in preparation for logical replication.
> + */
> +static bool
> +setup_publisher(LogicalRepInfo *dbinfo)
> ```
>
> But this function verifies the source server. I felt they should be in the
> different function.
I split setup_publisher() into check_publisher() and setup_publisher().
> 05.
> ```
> +/*
> + * Is the target server ready for logical replication?
> + */
> +static bool
> +setup_subscriber(LogicalRepInfo *dbinfo)
> ````
>
> Actually, this function does not set up subscriber. It just verifies whether the
> target can become a subscriber, right? If should be renamed.
I renamed setup_subscriber() -> check_subscriber().
> 06.
> ```
> + atexit(cleanup_objects_atexit);
> ```
>
> The registration of the cleanup function is too early. This sometimes triggers
> a core-dump. E.g.,
I forgot to apply the atexit() patch.
> 07.
> Missing a removal of publications on the standby.
It was on my list to do. It will be in the next patch.
> 08.
> Missing registration of LogicalRepInfo in the typedefs.list.
Done.
> 09
> ```
> + <refsynopsisdiv>
> + <cmdsynopsis>
> + <command>pg_subscriber</command>
> + <arg rep="repeat"><replaceable>option</replaceable></arg>
> + </cmdsynopsis>
> + </refsynopsisdiv>
> ```
>
> Can you reply my comment#2 [4]? I think mandatory options should be written.
I included the mandatory options into the synopsis.
> 10.
> Just to confirm - will you implement start_standby/stop_standby functions in next version?
It is still on my list.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-29 09:57 ` Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 0 replies; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-29 09:57 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
Dear Euler,
>
It is still on my list. As I said in a previous email I'm having a hard time
reviewing pieces from your 0002 patch because you include a bunch of things
...
It is still on my list.
>
I understood that patches we posted were bad. Sorry for inconvenience.
So I extracted changes only related with them. Can you review them and include
If it seems OK?
v10-0001: same as v9-0001.
0002: not related with our changes, but this fixed a small bug.
The third argument of getopt_long was not correct.
0003: adds start_standby/stop_standby funcitons
0004: tries to refactor some structures.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
Attachments:
[application/octet-stream] v10-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch (73.2K, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v10-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch)
download | inline diff:
From 8eff27951f4b778639e91e0a02f871628af60afd Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v10 1/4] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 305 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1805 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2322 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..99d4fcee49
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,305 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..cb97dbda5e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1805 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Is the source server ready for logical replication? If so, create the
+ * publications and replication slots in preparation for logical replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the target server ready for logical replication?
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!setup_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication and
+ * create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.43.0
[application/octet-stream] v10-0002-fix-getopt-options.patch (761B, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v10-0002-fix-getopt-options.patch)
download | inline diff:
From 722a2ba9bb647559def5f6d826fe9886973910d4 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 08:43:11 +0000
Subject: [PATCH v10 2/4] fix getopt options
---
src/bin/pg_basebackup/pg_subscriber.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index cb97dbda5e..df770f0fc8 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1448,7 +1448,7 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
long_options, &option_index)) != -1)
{
switch (c)
--
2.43.0
[application/octet-stream] v10-0003-add-start_standby-stop_standby-functions.patch (4.6K, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v10-0003-add-start_standby-stop_standby-functions.patch)
download | inline diff:
From e17e392b68011f9173ddaa9bc88009a8e5629c44 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 08:27:48 +0000
Subject: [PATCH v10 3/4] add start_standby/stop_standby functions
---
src/bin/pg_basebackup/pg_subscriber.c | 104 +++++++++++++++++---------
1 file changed, 69 insertions(+), 35 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index df770f0fc8..f535e7160f 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -75,6 +75,9 @@ static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void start_standby(char *server_start_log);
+static void stop_standby(void);
+
#define USEC_PER_SEC 1000000
#define WAIT_INTERVAL 1 /* 1 second */
@@ -1363,6 +1366,68 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+/*
+ * Start a standby server with given logfile. This also decides filename if not
+ * determined yet. If the start is failed, the process exits with error
+ * messages.
+ */
+static void
+start_standby(char *server_start_log)
+{
+ int rc;
+ char *pg_ctl_cmd;
+
+ Assert(server_start_log != NULL);
+
+ if (server_start_log[0] == '\0')
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", subscriber_dir,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+
+ pg_log_info("starting the standby server");
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path,
+ subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+/*
+ * Stop a standby server. If the stop is failed, the process exits with error
+ * messages.
+ */
+static void
+stop_standby(void)
+{
+ int rc;
+ char *pg_ctl_cmd;
+
+ pg_log_info("stopping the standby server");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path,
+ subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+}
+
int
main(int argc, char **argv)
{
@@ -1383,16 +1448,10 @@ main(int argc, char **argv)
int c;
int option_index;
- int rc;
-
- char *pg_ctl_cmd;
char *base_dir;
- char *server_start_log;
+ char server_start_log[MAXPGPATH] = {0};
- char timebuf[128];
- struct timeval time;
- time_t tt;
int len;
char *pub_base_conninfo = NULL;
@@ -1638,9 +1697,7 @@ main(int argc, char **argv)
pg_log_info("standby is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 0);
+ stop_standby();
}
else
{
@@ -1705,26 +1762,7 @@ main(int argc, char **argv)
/*
* Start subscriber and wait until accepting connections.
*/
- pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby(server_start_log);
/*
* Waiting the subscriber to be promoted.
@@ -1779,11 +1817,7 @@ main(int argc, char **argv)
/*
* Stop the subscriber.
*/
- pg_log_info("stopping the subscriber");
-
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 0);
+ stop_standby();
/*
* Change system identifier.
--
2.43.0
[application/octet-stream] v10-0004-Divide-LogicalReplInfo-into-some-strcutures.patch (48.0K, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/5-v10-0004-Divide-LogicalReplInfo-into-some-strcutures.patch)
download | inline diff:
From afd8eb318c40e4602268cf5a82b209cb7fa07af2 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 07:03:59 +0000
Subject: [PATCH v10 4/4] Divide LogicalReplInfo into some strcutures
---
src/bin/pg_basebackup/pg_subscriber.c | 674 ++++++++++++++------------
src/tools/pgindent/typedefs.list | 4 +
2 files changed, 374 insertions(+), 304 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index f535e7160f..a8df8d5135 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -32,51 +32,72 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct LogicalRepPerdbInfoArr
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+ bool made_transient_replslot;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ char *server_log;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
static char *get_base_conninfo(char *conninfo, char *dbname,
const char *noderole);
-static bool get_exec_path(const char *path);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static bool setup_publisher(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_sysid_for_standby(StandbyInfo *standby);
+static void modify_sysid(const char *bindir, const char *datadir);
+static bool setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-
-static void start_standby(char *server_start_log);
-static void stop_standby(void);
+static void wait_for_end_recovery(StandbyInfo *standby,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+
+static void start_standby(StandbyInfo *standby);
+static void stop_standby(StandbyInfo *standby);
#define USEC_PER_SEC 1000000
#define WAIT_INTERVAL 1 /* 1 second */
@@ -84,25 +105,18 @@ static void stop_standby(void);
/* Options */
static const char *progname;
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
static bool dry_run = false;
static bool retain = false;
static int recovery_timeout = 0;
static bool success = false;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
-
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
enum WaitPMResult
{
@@ -112,6 +126,30 @@ enum WaitPMResult
POSTMASTER_FAILED
};
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
+
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
+{
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
+
/*
* Cleanup objects that were created by pg_subscriber if there is an error.
@@ -129,38 +167,51 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
- disconnect_database(conn);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
}
}
}
- if (made_transient_replslot)
+ if (primary.made_transient_replslot)
{
- conn = connect_database(dbinfo[0].pubconninfo);
+ char transient_replslot[NAMEDATALEN];
+
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+
if (conn != NULL)
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+
+ drop_replication_slot(conn, &dbarr.perdb[0], transient_replslot);
disconnect_database(conn);
}
}
@@ -246,15 +297,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
- int rc;
+ int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -279,30 +331,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
-
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -366,49 +400,35 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases.
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
- SimpleStringListCell *cell;
- int i = 0;
+ SimpleStringListCell *cell;
+ int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ dbarr.perdb = (LogicalRepPerdbInfo *) pg_malloc0(ndbs *
+ sizeof(LogicalRepPerdbInfo));
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -426,6 +446,8 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
return conn;
}
@@ -438,19 +460,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -473,43 +494,40 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is " UINT64_FORMAT " on publisher",
+ primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_sysid_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is " UINT64_FORMAT " on subscriber",
+ standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -518,7 +536,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -553,7 +571,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -574,7 +592,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* publications and replication slots in preparation for logical replication.
*/
static bool
-setup_publisher(LogicalRepInfo *dbinfo)
+setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -597,7 +615,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
* max_replication_slots >= current + number of dbs to be converted
* max_wal_senders >= current + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -635,10 +653,11 @@ setup_publisher(LogicalRepInfo *dbinfo)
* use after the transformation, hence, it will be removed at the end of
* this process.
*/
- if (primary_slot_name)
+ if (standby.primary_slot_name)
{
appendPQExpBuffer(str,
- "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'",
+ standby.primary_slot_name);
pg_log_debug("command is: %s", str->data);
@@ -653,13 +672,14 @@ setup_publisher(LogicalRepInfo *dbinfo)
{
pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
PQntuples(res), 1);
- pg_free(primary_slot_name); /* it is not being used. */
- primary_slot_name = NULL;
+ pg_free(standby.primary_slot_name); /* it is not being used. */
+ standby.primary_slot_name = NULL;
return false;
}
else
{
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
}
PQclear(res);
@@ -673,26 +693,29 @@ setup_publisher(LogicalRepInfo *dbinfo)
return false;
}
- if (max_repslots - cur_repslots < num_dbs)
+ if (max_repslots - cur_repslots < dbarr->ndbs)
{
- pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ pg_log_error("publisher requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ cur_repslots + dbarr->ndbs);
return false;
}
- if (max_walsenders - cur_walsenders < num_dbs)
+ if (max_walsenders - cur_walsenders < dbarr->ndbs)
{
- pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
- pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain",
+ dbarr->ndbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.",
+ cur_walsenders + dbarr->ndbs);
return false;
}
- for (int i = 0; i < num_dbs; i++)
+ for (int i = 0; i < dbarr->ndbs; i++)
{
- char pubname[NAMEDATALEN];
- char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -712,43 +735,21 @@ setup_publisher(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
/*
* Create publication on publisher. This step should be executed
* *before* promoting the subscriber to avoid any transactions between
* consistent LSN and the new publication rows (such transactions
* wouldn't see the new publication rows resulting in an error).
*/
- create_publication(conn, &dbinfo[i]);
-
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ create_publication(conn, primary, perdb);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL &&
+ !dry_run)
return false;
disconnect_database(conn);
@@ -761,7 +762,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
* Is the target server ready for logical replication?
*/
static bool
-setup_subscriber(LogicalRepInfo *dbinfo)
+setup_subscriber(StandbyInfo *standby, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -781,7 +782,7 @@ setup_subscriber(LogicalRepInfo *dbinfo)
* max_logical_replication_workers >= number of dbs to be converted
* max_worker_processes >= 1 + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -798,35 +799,41 @@ setup_subscriber(LogicalRepInfo *dbinfo)
max_repslots = atoi(PQgetvalue(res, 1, 0));
max_wprocs = atoi(PQgetvalue(res, 2, 0));
if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
- primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+ standby->primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
- pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+ pg_log_debug("subscriber: primary_slot_name: %s", standby->primary_slot_name);
PQclear(res);
disconnect_database(conn);
- if (max_repslots < num_dbs)
+ if (max_repslots < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_lrworkers < num_dbs)
+ if (max_lrworkers < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
- pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain",
+ dbarr->ndbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_wprocs < num_dbs + 1)
+ if (max_wprocs < dbarr->ndbs + 1)
{
- pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
- pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ pg_log_error("subscriber requires %d worker processes, but only %d remain",
+ dbarr->ndbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.",
+ dbarr->ndbs + 1);
return false;
}
@@ -841,28 +848,32 @@ setup_subscriber(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (temporary)
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
+ else
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name,
+ NAMEDATALEN);
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
@@ -874,17 +885,19 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
+
+ pg_log_info("create replication slot \"%s\" on publisher", slot_name);
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
+ if (temporary)
+ primary.made_transient_replslot = true;
else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -898,14 +911,16 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -915,7 +930,8 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -968,19 +984,17 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* the recovery process. By default, it waits forever.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(StandbyInfo *standby,
+ const char *dbname)
{
PGconn *conn;
PGresult *res;
int status = POSTMASTER_STILL_STARTING;
int timer = 0;
- char *pg_ctl_cmd;
- int rc;
-
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(standby->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -1021,9 +1035,13 @@ wait_for_end_recovery(const char *conninfo)
*/
if (recovery_timeout > 0 && timer >= recovery_timeout)
{
+ char *pg_ctl_cmd;
+ int rc;
+
pg_log_error("recovery timed out");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby->bindir, standby->pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
@@ -1051,17 +1069,22 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -1081,7 +1104,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -1094,7 +1117,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -1105,9 +1128,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -1117,14 +1140,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -1136,24 +1159,28 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ pg_log_info("dropping publication \"%s\" on database \"%s\"",
+ pubname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
- pg_log_debug("command is: %s", str->data);
if (!dry_run)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s",
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1174,19 +1201,30 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname,
+ concat_conninfo_dbname(primary->base_conninfo,
+ perdb->dbname),
+ pubname);
pg_log_debug("command is: %s", str->data);
@@ -1196,14 +1234,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1215,16 +1253,20 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"",
+ subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1232,7 +1274,8 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s",
+ subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1251,18 +1294,23 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'",
+ subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1303,7 +1351,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1317,7 +1365,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1336,16 +1384,19 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1354,7 +1405,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1372,20 +1423,20 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* messages.
*/
static void
-start_standby(char *server_start_log)
+start_standby(StandbyInfo *standby)
{
int rc;
char *pg_ctl_cmd;
- Assert(server_start_log != NULL);
-
- if (server_start_log[0] == '\0')
+ if (standby->server_log == NULL)
{
char timebuf[128];
struct timeval time;
time_t tt;
int len;
+ standby->server_log = (char *) pg_malloc0(MAXPGPATH);
+
/* append timestamp with ISO 8601 format. */
gettimeofday(&time, NULL);
tt = (time_t) time.tv_sec;
@@ -1393,8 +1444,8 @@ start_standby(char *server_start_log)
snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
".%03d", (int) (time.tv_usec / 1000));
- len = snprintf(server_start_log, MAXPGPATH,
- "%s/%s/server_start_%s.log", subscriber_dir,
+ len = snprintf(standby->server_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
PGS_OUTPUT_DIR, timebuf);
if (len >= MAXPGPATH)
{
@@ -1404,8 +1455,8 @@ start_standby(char *server_start_log)
}
pg_log_info("starting the standby server");
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path,
- subscriber_dir, server_start_log);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -l \"%s\"",
+ standby->bindir, standby->pgdata, standby->server_log);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 1);
}
@@ -1415,15 +1466,15 @@ start_standby(char *server_start_log)
* messages.
*/
static void
-stop_standby(void)
+stop_standby(StandbyInfo *standby)
{
int rc;
char *pg_ctl_cmd;
pg_log_info("stopping the standby server");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path,
- subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s", standby->bindir,
+ standby->pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1454,12 +1505,8 @@ main(int argc, char **argv)
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
struct stat statbuf;
PGconn *conn;
@@ -1513,7 +1560,7 @@ main(int argc, char **argv)
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
@@ -1526,7 +1573,7 @@ main(int argc, char **argv)
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
case 'n':
@@ -1562,7 +1609,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1585,9 +1632,10 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo,
+ "publisher");
+ if (primary.base_conninfo == NULL)
exit(1);
if (sub_conninfo_str == NULL)
@@ -1596,8 +1644,12 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+
+ standby.base_conninfo = get_base_conninfo(sub_conninfo_str, NULL,
+ "subscriber");
+
+
+ if (standby.base_conninfo == NULL)
exit(1);
if (database_names.head == NULL)
@@ -1612,7 +1664,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1628,23 +1680,23 @@ main(int argc, char **argv)
/*
* Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_sysid_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1654,7 +1706,7 @@ main(int argc, char **argv)
* Create the output directory to store any data generated by this tool.
*/
base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1668,7 +1720,7 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* The standby server must be running. That's because some checks will be
@@ -1681,7 +1733,7 @@ main(int argc, char **argv)
/*
* Check if the standby server is ready for logical replication.
*/
- if (!setup_subscriber(dbinfo))
+ if (!setup_subscriber(&standby, &dbarr))
exit(1);
/*
@@ -1691,13 +1743,13 @@ main(int argc, char **argv)
* if the primary slot is in use. We could use an extra connection for
* it but it doesn't seem worth.
*/
- if (!setup_publisher(dbinfo))
+ if (!setup_publisher(&primary, &dbarr))
exit(1);
pg_log_info("standby is up and running");
pg_log_info("stopping the server to start the transformation steps");
- stop_standby();
+ stop_standby(&standby);
}
else
{
@@ -1723,11 +1775,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true, &dbarr.perdb[0]);
+
/*
* Write recovery parameters.
@@ -1753,7 +1805,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1762,32 +1814,32 @@ main(int argc, char **argv)
/*
* Start subscriber and wait until accepting connections.
*/
- start_standby(server_start_log);
+ start_standby(&standby);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(&standby, dbarr.perdb[0].dbname);
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, &primary, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
-
- disconnect_database(conn);
+ enable_subscription(conn, perdb);
}
/*
@@ -1798,31 +1850,45 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- if (primary_slot_name != NULL)
- pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ char transient_replslot[NAMEDATALEN];
+
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+
+ if (standby.primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary",
+ standby.primary_slot_name);
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher",
+ transient_replslot);
pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+ char transient_replslot[NAMEDATALEN];
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
+
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ drop_replication_slot(conn, perdb, transient_replslot);
disconnect_database(conn);
}
/*
* Stop the subscriber.
*/
- stop_standby();
+ stop_standby(&standby);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* The log file is kept if retain option is specified or this tool does
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7e866e3c3d..e9e3db9ffc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1506,6 +1506,8 @@ LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
LogicalRepMsgType
LogicalRepPartMapEntry
+LogicalRepPerdbInfo
+LogicalRepPerdbInfoArr
LogicalRepPreparedTxnData
LogicalRepRelId
LogicalRepRelMapEntry
@@ -1884,6 +1886,7 @@ PREDICATELOCK
PREDICATELOCKTAG
PREDICATELOCKTARGET
PREDICATELOCKTARGETTAG
+PrimaryInfo
PROCESS_INFORMATION
PROCLOCK
PROCLOCKTAG
@@ -2460,6 +2463,7 @@ SQLValueFunctionOp
SSL
SSLExtensionInfoContext
SSL_CTX
+StandbyInfo
STARTUPINFO
STRLEN
SV
--
2.43.0
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-30 02:01 ` Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 1 reply; 87+ messages in thread
From: Euler Taveira @ 2024-01-30 02:01 UTC (permalink / raw)
To: [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Sun, Jan 28, 2024, at 10:10 PM, Euler Taveira wrote:
> On Fri, Jan 26, 2024, at 4:55 AM, Hayato Kuroda (Fujitsu) wrote:
>> Again, thanks for updating the patch! There are my random comments for v9.
>
> Thanks for checking v9. I already incorporated some of the points below into
> the next patch. Give me a couple of hours to include some important points.
Here it is another patch that includes the following changes:
* rename the tool to pg_createsubscriber: it was the name with the most votes
[1].
* fix recovery-timeout option [2]
* refactor: split setup_publisher into check_publisher (that contains only GUC
checks) and setup_publisher (that does what the name suggests) [2]
* doc: verbose option can be specified multiple times [2]
* typedefs.list: add LogicalRepInfo [2]
* fix: register cleanup routine after the data structure (dbinfo) is assigned
[2]
* doc: add mandatory options to synopsis [3]
* refactor: rename some options (such as publisher-conninfo to
publisher-server) to use the same pattern as pg_rewind.
* feat: remove publications from subscriber [2]
* feat: use temporary replication slot to get consistent LSN [3]
* refactor: move subscriber setup to its own function
* refactor: stop standby server to its own function [2]
* refactor: start standby server to its own function [2]
* fix: getopt options [4]
There is a few open items in my list. I included v10-0002. I had already
included a refactor to include start/stop functions so I didn't include
v10-0003. I'll check v10-0004 tomorrow.
One open item that is worrying me is how to handle the pg_ctl timeout. This
patch does nothing and the user should use PGCTLTIMEOUT environment variable to
avoid that the execution is canceled after 60 seconds (default for pg_ctl).
Even if you set a high value, it might not be enough for cases like
time-delayed replica. Maybe pg_ctl should accept no timeout as --timeout
option. I'll include this caveat into the documentation but I'm afraid it is
not sufficient and we should provide a better way to handle this situation.
[1] https://www.postgresql.org/message-id/b315c7da-7ab1-4014-a2a9-8ab6ae26017c%40app.fastmail.com
[2] https://www.postgresql.org/message-id/TY3PR01MB98895A551923953B3DA3C7C8F5792%40TY3PR01MB9889.jpnprd0...
[3] https://www.postgresql.org/message-id/[email protected]....
[4] https://www.postgresql.org/message-id/TY3PR01MB98891E7735141FE8760CEC4AF57E2%40TY3PR01MB9889.jpnprd0...
--
Euler Taveira
EDB https://www.enterprisedb.com/
Attachments:
[text/x-patch] v11-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch (76.2K, ../../[email protected]/3-v11-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch)
download | inline diff:
From 67c6e63cf34885bdd687c06e1e071d15f42cdf2a Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v11] Creates a new logical replica from a standby server
A new tool called pg_createsubscriber can convert a physical replica
into a logical replica. It runs on the target server and should be able
to connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Stop the
target server. Change the system identifier from the target server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_createsubscriber.sgml | 322 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_createsubscriber.c | 1852 +++++++++++++++++
.../t/040_pg_createsubscriber.pl | 44 +
.../t/041_pg_createsubscriber_standby.pl | 139 ++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 2387 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_createsubscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_createsubscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..a2b5eea0e0 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgCreateSubscriber SYSTEM "pg_createsubscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
new file mode 100644
index 0000000000..1c78ff92e0
--- /dev/null
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -0,0 +1,322 @@
+<!--
+doc/src/sgml/ref/pg_createsubscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgcreatesubscriber">
+ <indexterm zone="app-pgcreatesubscriber">
+ <primary>pg_createsubscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_createsubscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_createsubscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_createsubscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ <group choice="plain">
+ <group choice="req">
+ <arg choice="plain"><option>-D</option> </arg>
+ <arg choice="plain"><option>--pgdata</option></arg>
+ </group>
+ <replaceable>datadir</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-P</option></arg>
+ <arg choice="plain"><option>--publisher-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><option>--subscriber-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-d</option></arg>
+ <arg choice="plain"><option>--database</option></arg>
+ </group>
+ <replaceable>dbname</replaceable>
+ </group>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_createsubscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_createsubscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_createsubscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--publisher-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--subscriber-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--recovery-timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_createsubscriber</application> to output progress messages
+ and detailed information about each step to standard error.
+ Repeating the option causes additional debug-level messages to appear on
+ standard error.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_createsubscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_createsubscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_createsubscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_createsubscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. A temporary
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_createsubscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_createsubscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_createsubscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_createsubscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..c5edd244ef 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgCreateSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..b3a6f5a2fe 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_createsubscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..ded434b683 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_createsubscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_createsubscriber: $(WIN32RES) pg_createsubscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_createsubscriber$(X) '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_createsubscriber$(X) pg_createsubscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..345a2d6fcd 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_createsubscriber_sources = files(
+ 'pg_createsubscriber.c'
+)
+
+if host_system == 'windows'
+ pg_createsubscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_createsubscriber',
+ '--FILEDESC', 'pg_createsubscriber - create a new logical replica from a standby server',])
+endif
+
+pg_createsubscriber = executable('pg_createsubscriber',
+ pg_createsubscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_createsubscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_createsubscriber.pl',
+ 't/041_pg_createsubscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
new file mode 100644
index 0000000000..478560b3e4
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -0,0 +1,1852 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_createsubscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_createsubscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_createsubscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool check_publisher(LogicalRepInfo *dbinfo);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static bool check_subscriber(LogicalRepInfo *dbinfo);
+static bool setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *server_logfile_name(const char *datadir);
+static void start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile);
+static void stop_standby_server(const char *pg_ctl_path, const char *datadir);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_createsubscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-server=CONNSTR publisher connection string\n"));
+ printf(_(" -S, --subscriber-server=CONNSTR subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Create the publications and replication slots in preparation for logical
+ * replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 31 characters (20 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_createsubscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 42
+ * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_createsubscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the primary server ready for logical replication?
+ */
+static bool
+check_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Is the standby server ready for logical replication?
+ */
+static bool
+check_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create the subscriptions, adjust the initial location for logical replication and
+ * enable the subscriptions. That's the last step for logical repliation setup.
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
+{
+ PGconn *conn;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Since the publication was created before the consistent LSN, it is
+ * available on the subscriber when the physical replica is promoted.
+ * Remove publications from the subscriber because it has no use.
+ */
+ drop_publication(conn, &dbinfo[i]);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_createsubscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ if (transient_replslot)
+ appendPQExpBufferStr(str, " TEMPORARY");
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (!transient_replslot)
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+static char *
+server_logfile_name(const char *datadir)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ char *filename;
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ filename = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(filename, MAXPGPATH, "%s/%s/server_start_%s.log", datadir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ return filename;
+}
+
+static void
+start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, datadir, logfile);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static void
+stop_standby_server(const char *pg_ctl_path, const char *datadir)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, datadir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_createsubscriber must have created
+ * this publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_createsubscriber_ prefix followed by the
+ * exact database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-server", required_argument, NULL, 'P'},
+ {"subscriber-server", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+
+ char *base_dir;
+ char *server_start_log;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+ char temp_replslot[NAMEDATALEN] = {0};
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_createsubscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_createsubscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /* Register a function to clean up objects in case of failure. */
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ server_start_log = server_logfile_name(subscriber_dir);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!check_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication. This
+ * routine checks if a replication slot is in use on primary so it
+ * relies on check_subscriber() to obtain the primary_slot_name.
+ * That's why it is called after it.
+ */
+ if (!check_publisher(dbinfo))
+ exit(1);
+
+ /*
+ * Create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ /* Stop the standby server. */
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a temporary logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. It is ok to use a temporary replication slot here
+ * because it will have a short lifetime and it is only used as a mark to
+ * start the logical replication.
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+ start_standby_server(pg_ctl_path, subscriber_dir, server_start_log);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create the subscription for each database on subscriber. It does not
+ * enable it immediately because it needs to adjust the logical
+ * replication start point to the LSN reported by consistent_lsn (see
+ * set_replication_progress). It also cleans up publications created by
+ * this tool and replication to the standby.
+ */
+ if (!setup_subscriber(dbinfo, consistent_lsn))
+ exit(1);
+
+ /*
+ * If the primary_slot_name exists on primary, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops this replication slot later.
+ */
+ if (primary_slot_name != NULL)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ }
+ else
+ {
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
new file mode 100644
index 0000000000..0f02b1bfac
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_createsubscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_createsubscriber');
+program_version_ok('pg_createsubscriber');
+program_options_handling_ok('pg_createsubscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_createsubscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres',
+ '--subscriber-server', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
new file mode 100644
index 0000000000..534bc53a76
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_createsubscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_createsubscriber on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_createsubscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 90b37b919c..2c5725e18d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1505,6 +1505,7 @@ LogicalRepBeginData
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
+LogicalRepInfo
LogicalRepMsgType
LogicalRepPartMapEntry
LogicalRepPreparedTxnData
--
2.30.2
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-30 09:26 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-30 09:26 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
Dear Euler,
Thanks for updating the patch!
> One open item that is worrying me is how to handle the pg_ctl timeout. This
> patch does nothing and the user should use PGCTLTIMEOUT environment variable to
> avoid that the execution is canceled after 60 seconds (default for pg_ctl).
> Even if you set a high value, it might not be enough for cases like
> time-delayed replica. Maybe pg_ctl should accept no timeout as --timeout
> option. I'll include this caveat into the documentation but I'm afraid it is
> not sufficient and we should provide a better way to handle this situation.
I felt you might be confused a bit. Even if the recovery_min_apply_delay is set,
e.g., 10h., the pg_ctl can start and stop the server. This is because the
walreceiver serialize changes as soon as they received. The delay is done by the
startup process. There are no unflushed data, so server instance can be turned off.
If you meant the combination of recovery-timeout and time-delayed replica, yes,
it would be likely to occur. But in the case, using like --no-timeout option is
dangerous. I think we should overwrite recovery_min_apply_delay to zero. Thought?
Below part contains my comments for v11-0001. Note that the ordering is random.
01. doc
```
<group choice="req">
<arg choice="plain"><option>-D</option> </arg>
<arg choice="plain"><option>--pgdata</option></arg>
</group>
```
According to other documentation like pg_upgrade, we do not write both longer
and shorter options in the synopsis section.
02. doc
```
<para>
<application>pg_createsubscriber</application> takes the publisher and subscriber
connection strings, a cluster directory from a physical replica and a list of
database names and it sets up a new logical replica using the physical
recovery process.
</para>
```
I found that you did not include my suggestion without saying [1]. Do you dislike
the comment or still considering?
03. doc
```
<term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
```
Too many blank after -P.
04. doc
```
<para>
<application>pg_createsubscriber</application> checks if the target data
directory is used by a physical replica. Stop the physical replica if it is
running. One of the next steps is to add some recovery parameters that
requires a server start. This step avoids an error.
</para>
```
I think doc is not updated. Currently (not sure it is good) the physical replica
must be running before doing pg_createsubscriber.
05. doc
```
each specified database on the source server. The replication slot name
contains a <literal>pg_createsubscriber</literal> prefix. These replication
slots will be used by the subscriptions in a future step. A temporary
```
According to the "Replication Slot Management" subsection in logical-replication.sgml,
the format of names should be described expressly, e.g.:
```
These replication slots have generated names:"pg_createsubscriber_%u_%d" (parameters: Subscription oid, Process id pid).
```
Publications, a temporary slot, and subscriptions should be described as well.
06. doc
```
Next, <application>pg_createsubscriber</application> creates one publication
for each specified database on the source server. Each publication
replicates changes for all tables in the database. The publication name
```
Missing update. Publications are created before creating replication slots.
07. general
I think there are some commenting conversions in PG, but this file breaks it.
* Comments must be one-line as much as possible
* Periods must not be added when the comment is one-line.
* Initial character must be capital.
08. general
Some pg_log_error() + exit(1) can be replaced with pg_fatal().
09. LogicalRepInfo
```
char *subconninfo; /* subscription connection string for logical
* replication */
```
As I posted in comment#8[2], I don't think it is "subscription connection". Also,
"for logical replication" is bit misreading because it would not be passed to
workers.
10. get_base_conninfo
```
static char *
get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
...
/*
* If --database option is not provided, try to obtain the dbname from
* the publisher conninfo. If dbname parameter is not available, error
* out.
*/
```
I'm not sure getting dbname from the conninfo improves user-experience. I felt
it may trigger an unintended targeting.
(I still think the publisher-server should be removed)
11. check_data_directory
```
/*
* Is it a cluster directory? These are preliminary checks. It is far from
* making an accurate check. If it is not a clone from the publisher, it will
* eventually fail in a future step.
*/
static bool
check_data_directory(const char *datadir)
```
We shoud also check whether pg_createsubscriber can create a file and a directory.
GetDataDirectoryCreatePerm() verifies it.
12. main
```
/* Register a function to clean up objects in case of failure. */
atexit(cleanup_objects_atexit);
```
According to the manpage, callback functions would not be called when it exits
due to signals:
> Functions registered using atexit() (and on_exit(3)) are not called if a
> process terminates abnormally because of the delivery of a signal.
Do you have a good way to handle the case? One solution is to output created
objects in any log level, but the consideration may be too much. Thought?
13, main
```
/*
* Create a temporary logical replication slot to get a consistent LSN.
```
Just to clarify - I still think the process exits before here in case of dry run.
In case of pg_resetwal, the process exits before doing actual works like
RewriteControlFile().
14. main
```
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops this replication slot later.
```
But there are possibilities to exit(1) in drop_replication_slot(). Is it acceptable?
15. wait_for_end_recovery
```
/*
* Bail out after recovery_timeout seconds if this option is set.
*/
if (recovery_timeout > 0 && timer >= recovery_timeout)
```
Hmm, IIUC, it should be enabled by default [3]. Do you have anything in your mind?
16. main
```
/*
* Create the subscription for each database on subscriber. It does not
* enable it immediately because it needs to adjust the logical
* replication start point to the LSN reported by consistent_lsn (see
* set_replication_progress). It also cleans up publications created by
* this tool and replication to the standby.
*/
if (!setup_subscriber(dbinfo, consistent_lsn))
```
Subscriptions would be created and replication origin would be moved forward here,
but latter one can be done only by the superuser. I felt that this should be
checked in check_subscriber().
17. main
```
/*
* Change system identifier.
*/
modify_sysid(pg_resetwal_path, subscriber_dir);
```
Even if I executed without -v option, an output from pg_resetwal command appears.
It seems bit strange.
```
$ pg_createsubscriber -D data_N2/ -P "port=5431 user=postgres" -S "port=5432 user=postgres" -d postgres
Write-ahead log reset
$
```
[1]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[2]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[3]: https://www.postgresql.org/message-id/CAA4eK1JRgnhv_ySzuFjN7UaX9qxz5Hqcwew7Fk%3D%2BAbJbu0Kd9w%40mail...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
^ permalink raw reply [nested|flat] 87+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-31 12:52 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-31 12:52 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>
Dear Euler,
I extracted some review comments which may require many efforts. I hope this makes them
easy to review.
0001: not changed from yours.
0002: avoid to use replication connections. Source: comment #3[1]
0003: Remove -P option and use primary_conninfo instead. Source: [2]
0004: Exit earlier when dry_run is specified. Source: [3]
0005: Refactor data structures. Source: [4]
[1]: https://www.postgresql.org/message-id/TY3PR01MB9889593399165B9A04106741F5662%40TY3PR01MB9889.jpnprd0...
[2]: https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
[3]: https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
[4]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
[application/octet-stream] v12-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch (76.2K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v12-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch)
download | inline diff:
From 14e74d8df45fcc25d0ba9d8a8417a1c9bb95b6aa Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v12 1/5] Creates a new logical replica from a standby server
A new tool called pg_createsubscriber can convert a physical replica
into a logical replica. It runs on the target server and should be able
to connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Stop the
target server. Change the system identifier from the target server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_createsubscriber.sgml | 322 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_createsubscriber.c | 1852 +++++++++++++++++
.../t/040_pg_createsubscriber.pl | 44 +
.../t/041_pg_createsubscriber_standby.pl | 139 ++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 2387 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_createsubscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_createsubscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..a2b5eea0e0 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgCreateSubscriber SYSTEM "pg_createsubscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
new file mode 100644
index 0000000000..1c78ff92e0
--- /dev/null
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -0,0 +1,322 @@
+<!--
+doc/src/sgml/ref/pg_createsubscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgcreatesubscriber">
+ <indexterm zone="app-pgcreatesubscriber">
+ <primary>pg_createsubscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_createsubscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_createsubscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_createsubscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ <group choice="plain">
+ <group choice="req">
+ <arg choice="plain"><option>-D</option> </arg>
+ <arg choice="plain"><option>--pgdata</option></arg>
+ </group>
+ <replaceable>datadir</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-P</option></arg>
+ <arg choice="plain"><option>--publisher-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><option>--subscriber-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-d</option></arg>
+ <arg choice="plain"><option>--database</option></arg>
+ </group>
+ <replaceable>dbname</replaceable>
+ </group>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_createsubscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_createsubscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_createsubscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--publisher-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--subscriber-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--recovery-timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_createsubscriber</application> to output progress messages
+ and detailed information about each step to standard error.
+ Repeating the option causes additional debug-level messages to appear on
+ standard error.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_createsubscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_createsubscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_createsubscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_createsubscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. A temporary
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_createsubscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_createsubscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_createsubscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_createsubscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..c5edd244ef 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgCreateSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..b3a6f5a2fe 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_createsubscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..ded434b683 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_createsubscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_createsubscriber: $(WIN32RES) pg_createsubscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_createsubscriber$(X) '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_createsubscriber$(X) pg_createsubscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..345a2d6fcd 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_createsubscriber_sources = files(
+ 'pg_createsubscriber.c'
+)
+
+if host_system == 'windows'
+ pg_createsubscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_createsubscriber',
+ '--FILEDESC', 'pg_createsubscriber - create a new logical replica from a standby server',])
+endif
+
+pg_createsubscriber = executable('pg_createsubscriber',
+ pg_createsubscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_createsubscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_createsubscriber.pl',
+ 't/041_pg_createsubscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
new file mode 100644
index 0000000000..478560b3e4
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -0,0 +1,1852 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_createsubscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_createsubscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_createsubscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool check_publisher(LogicalRepInfo *dbinfo);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static bool check_subscriber(LogicalRepInfo *dbinfo);
+static bool setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *server_logfile_name(const char *datadir);
+static void start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile);
+static void stop_standby_server(const char *pg_ctl_path, const char *datadir);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_createsubscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-server=CONNSTR publisher connection string\n"));
+ printf(_(" -S, --subscriber-server=CONNSTR subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Create the publications and replication slots in preparation for logical
+ * replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 31 characters (20 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_createsubscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 42
+ * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_createsubscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the primary server ready for logical replication?
+ */
+static bool
+check_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Is the standby server ready for logical replication?
+ */
+static bool
+check_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create the subscriptions, adjust the initial location for logical replication and
+ * enable the subscriptions. That's the last step for logical repliation setup.
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
+{
+ PGconn *conn;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Since the publication was created before the consistent LSN, it is
+ * available on the subscriber when the physical replica is promoted.
+ * Remove publications from the subscriber because it has no use.
+ */
+ drop_publication(conn, &dbinfo[i]);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_createsubscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ if (transient_replslot)
+ appendPQExpBufferStr(str, " TEMPORARY");
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (!transient_replslot)
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+static char *
+server_logfile_name(const char *datadir)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ char *filename;
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ filename = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(filename, MAXPGPATH, "%s/%s/server_start_%s.log", datadir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ return filename;
+}
+
+static void
+start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, datadir, logfile);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static void
+stop_standby_server(const char *pg_ctl_path, const char *datadir)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, datadir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_createsubscriber must have created
+ * this publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_createsubscriber_ prefix followed by the
+ * exact database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-server", required_argument, NULL, 'P'},
+ {"subscriber-server", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+
+ char *base_dir;
+ char *server_start_log;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+ char temp_replslot[NAMEDATALEN] = {0};
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_createsubscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_createsubscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /* Register a function to clean up objects in case of failure. */
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ server_start_log = server_logfile_name(subscriber_dir);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!check_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication. This
+ * routine checks if a replication slot is in use on primary so it
+ * relies on check_subscriber() to obtain the primary_slot_name.
+ * That's why it is called after it.
+ */
+ if (!check_publisher(dbinfo))
+ exit(1);
+
+ /*
+ * Create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ /* Stop the standby server. */
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a temporary logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. It is ok to use a temporary replication slot here
+ * because it will have a short lifetime and it is only used as a mark to
+ * start the logical replication.
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+ start_standby_server(pg_ctl_path, subscriber_dir, server_start_log);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create the subscription for each database on subscriber. It does not
+ * enable it immediately because it needs to adjust the logical
+ * replication start point to the LSN reported by consistent_lsn (see
+ * set_replication_progress). It also cleans up publications created by
+ * this tool and replication to the standby.
+ */
+ if (!setup_subscriber(dbinfo, consistent_lsn))
+ exit(1);
+
+ /*
+ * If the primary_slot_name exists on primary, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops this replication slot later.
+ */
+ if (primary_slot_name != NULL)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ }
+ else
+ {
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
new file mode 100644
index 0000000000..0f02b1bfac
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_createsubscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_createsubscriber');
+program_version_ok('pg_createsubscriber');
+program_options_handling_ok('pg_createsubscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_createsubscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres',
+ '--subscriber-server', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
new file mode 100644
index 0000000000..534bc53a76
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_createsubscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_createsubscriber on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_createsubscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 91433d439b..f51f1ff23f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1505,6 +1505,7 @@ LogicalRepBeginData
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
+LogicalRepInfo
LogicalRepMsgType
LogicalRepPartMapEntry
LogicalRepPreparedTxnData
--
2.43.0
[application/octet-stream] v12-0002-Avoid-to-use-replication-connections.patch (4.1K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v12-0002-Avoid-to-use-replication-connections.patch)
download | inline diff:
From 36e626128b40a03ab7c50702fa91cb8acdd90e2f Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 31 Jan 2024 07:39:35 +0000
Subject: [PATCH v12 2/5] Avoid to use replication connections
---
doc/src/sgml/ref/pg_createsubscriber.sgml | 1 -
src/bin/pg_basebackup/pg_createsubscriber.c | 27 +++++++++------------
2 files changed, 11 insertions(+), 17 deletions(-)
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
index 1c78ff92e0..53b42e6161 100644
--- a/doc/src/sgml/ref/pg_createsubscriber.sgml
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -61,7 +61,6 @@ PostgreSQL documentation
The <application>pg_createsubscriber</application> should be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index 478560b3e4..47970b10d6 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -397,12 +397,8 @@ connect_database(const char *conninfo)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
- /* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
-
- conn = PQconnectdb(rconninfo);
+ conn = PQconnectdb(conninfo);
if (PQstatus(conn) != CONNECTION_OK)
{
pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
@@ -446,26 +442,26 @@ get_sysid_from_conn(const char *conninfo)
if (conn == NULL)
exit(1);
- res = PQexec(conn, "IDENTIFY_SYSTEM");
+ res = PQexec(conn, "SELECT * FROM pg_control_system();");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not send replication command \"%s\": %s",
+ pg_log_error("could not send command \"%s\": %s",
"IDENTIFY_SYSTEM", PQresultErrorMessage(res));
PQclear(res);
disconnect_database(conn);
exit(1);
}
- if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ if (PQntuples(res) != 1 || PQnfields(res) < 4)
{
pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
- PQntuples(res), PQnfields(res), 1, 3);
+ PQntuples(res), PQnfields(res), 1, 4);
PQclear(res);
disconnect_database(conn);
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ sysid = strtou64(PQgetvalue(res, 0, 2), NULL, 10);
pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
@@ -477,7 +473,7 @@ get_sysid_from_conn(const char *conninfo)
/*
* Obtain the system identifier from control file. It will be used to compare
* if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * and avoids a connection establishment.
*/
static uint64
get_control_from_datadir(const char *datadir)
@@ -905,10 +901,8 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
- appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
- if (transient_replslot)
- appendPQExpBufferStr(str, " TEMPORARY");
- appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+ appendPQExpBuffer(str, "SELECT * FROM pg_create_logical_replication_slot('%s', 'pgoutput', %s, false, false);",
+ slot_name, transient_replslot ? "true" : "false");
pg_log_debug("command is: %s", str->data);
@@ -948,7 +942,8 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
- appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBuffer(str, "SELECT * FROM pg_drop_replication_slot('%s');",
+ slot_name);
pg_log_debug("command is: %s", str->data);
--
2.43.0
[application/octet-stream] v12-0003-Remove-P-and-use-primary_conninfo-instead.patch (11.7K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v12-0003-Remove-P-and-use-primary_conninfo-instead.patch)
download | inline diff:
From 1315038f73179760000137cd34575645dc8fe86a Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 31 Jan 2024 09:20:54 +0000
Subject: [PATCH v12 3/5] Remove -P and use primary_conninfo instead
XXX: This may be a problematic when the OS user who started target instance is
not the current OS user and PGPASSWORD environment variable was used for
connecting to the primary server. In this case, the password would not be
written in the primary_conninfo and the PGPASSWORD variable might not be set.
This may lead an connection error. Is this a real issue? Note that using
PGPASSWORD is not recommended.
---
doc/src/sgml/ref/pg_createsubscriber.sgml | 17 +---
src/bin/pg_basebackup/pg_createsubscriber.c | 98 ++++++++++++-------
.../t/040_pg_createsubscriber.pl | 8 --
.../t/041_pg_createsubscriber_standby.pl | 5 +-
4 files changed, 65 insertions(+), 63 deletions(-)
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
index 53b42e6161..0abe1f6f28 100644
--- a/doc/src/sgml/ref/pg_createsubscriber.sgml
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -29,11 +29,6 @@ PostgreSQL documentation
<arg choice="plain"><option>--pgdata</option></arg>
</group>
<replaceable>datadir</replaceable>
- <group choice="req">
- <arg choice="plain"><option>-P</option></arg>
- <arg choice="plain"><option>--publisher-server</option></arg>
- </group>
- <replaceable>connstr</replaceable>
<group choice="req">
<arg choice="plain"><option>-S</option></arg>
<arg choice="plain"><option>--subscriber-server</option></arg>
@@ -83,16 +78,6 @@ PostgreSQL documentation
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
- <term><option>--publisher-server=<replaceable class="parameter">connstr</replaceable></option></term>
- <listitem>
- <para>
- The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>-S <replaceable class="parameter">connstr</replaceable></option></term>
<term><option>--subscriber-server=<replaceable class="parameter">connstr</replaceable></option></term>
@@ -304,7 +289,7 @@ PostgreSQL documentation
To create a logical replica for databases <literal>hr</literal> and
<literal>finance</literal> from a physical replica at <literal>foo</literal>:
<screen>
-<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -S "host=localhost" -d hr -d finance</userinput>
</screen>
</para>
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index 47970b10d6..f0e9db7793 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -51,10 +51,10 @@ typedef struct LogicalRepInfo
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
+static char *get_base_conninfo(char *conninfo, char *dbname);
static bool get_exec_path(const char *path);
static bool check_data_directory(const char *datadir);
+static char *get_primary_conninfo(const char *base_conninfo);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
static PGconn *connect_database(const char *conninfo);
@@ -88,7 +88,6 @@ static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
static const char *progname;
static char *subscriber_dir = NULL;
-static char *pub_conninfo_str = NULL;
static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
static char *primary_slot_name = NULL;
@@ -167,7 +166,6 @@ usage(void)
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nOptions:\n"));
printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
- printf(_(" -P, --publisher-server=CONNSTR publisher connection string\n"));
printf(_(" -S, --subscriber-server=CONNSTR subscriber connection string\n"));
printf(_(" -d, --database=DBNAME database to create a subscription\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
@@ -192,7 +190,7 @@ usage(void)
* dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -201,7 +199,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on subscriber");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -425,6 +423,58 @@ disconnect_database(PGconn *conn)
PQfinish(conn);
}
+/*
+ * Obtain primary_conninfo from the target server. The value would be used for
+ * connecting from the pg_createsubscriber itself and logical replication apply
+ * worker.
+ */
+static char *
+get_primary_conninfo(const char *base_conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ char *conninfo;
+ char *primaryconninfo;
+
+ pg_log_info("getting primary_conninfo from standby");
+
+ /*
+ * Construct a connection string to the target instance. Since dbinfo has
+ * not stored infomation yet, we must directly get the first element of the
+ * database list.
+ */
+ conninfo = concat_conninfo_dbname(base_conninfo, database_names.head->val);
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SHOW primary_conninfo;");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send command \"%s\": %s",
+ "SHOW primary_conninfo;", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ primaryconninfo = pg_strdup(PQgetvalue(res, 0, 0));
+
+ if (strlen(primaryconninfo) == 0)
+ {
+ pg_log_error("primary_conninfo is empty");
+ pg_log_error_hint("Check whether the target server is really a physical standby.");
+ exit(1);
+ }
+
+ pg_free(conninfo);
+ PQclear(res);
+ disconnect_database(conn);
+
+ return primaryconninfo;
+}
+
/*
* Obtain the system identifier using the provided connection. It will be used
* to compare if a data directory is a clone of another one.
@@ -1452,7 +1502,6 @@ main(int argc, char **argv)
{"help", no_argument, NULL, '?'},
{"version", no_argument, NULL, 'V'},
{"pgdata", required_argument, NULL, 'D'},
- {"publisher-server", required_argument, NULL, 'P'},
{"subscriber-server", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
@@ -1519,7 +1568,7 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
+ while ((c = getopt_long(argc, argv, "D:S:d:nrt:v",
long_options, &option_index)) != -1)
{
switch (c)
@@ -1527,9 +1576,6 @@ main(int argc, char **argv)
case 'D':
subscriber_dir = pg_strdup(optarg);
break;
- case 'P':
- pub_conninfo_str = pg_strdup(optarg);
- break;
case 'S':
sub_conninfo_str = pg_strdup(optarg);
break;
@@ -1581,34 +1627,13 @@ main(int argc, char **argv)
exit(1);
}
- /*
- * Parse connection string. Build a base connection string that might be
- * reused by multiple databases.
- */
- if (pub_conninfo_str == NULL)
- {
- /*
- * TODO use primary_conninfo (if available) from subscriber and
- * extract publisher connection string. Assume that there are
- * identical entries for physical and logical replication. If there is
- * not, we would fail anyway.
- */
- pg_log_error("no publisher connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
-
if (sub_conninfo_str == NULL)
{
pg_log_error("no subscriber connection string specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, dbname_conninfo);
if (sub_base_conninfo == NULL)
exit(1);
@@ -1618,7 +1643,7 @@ main(int argc, char **argv)
/*
* If --database option is not provided, try to obtain the dbname from
- * the publisher conninfo. If dbname parameter is not available, error
+ * the subscriber conninfo. If dbname parameter is not available, error
* out.
*/
if (dbname_conninfo)
@@ -1626,7 +1651,7 @@ main(int argc, char **argv)
simple_string_list_append(&database_names, dbname_conninfo);
num_dbs++;
- pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ pg_log_info("database \"%s\" was extracted from the subscriber connection string",
dbname_conninfo);
}
else
@@ -1637,6 +1662,9 @@ main(int argc, char **argv)
}
}
+ /* Obtain a connection string from the target */
+ pub_base_conninfo = get_primary_conninfo(sub_base_conninfo);
+
/*
* Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
*/
diff --git a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
index 0f02b1bfac..5c240a5417 100644
--- a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
@@ -17,18 +17,11 @@ my $datadir = PostgreSQL::Test::Utils::tempdir;
command_fails(['pg_createsubscriber'],
'no subscriber data directory specified');
-command_fails(
- [
- 'pg_createsubscriber',
- '--pgdata', $datadir
- ],
- 'no publisher connection string specified');
command_fails(
[
'pg_createsubscriber',
'--dry-run',
'--pgdata', $datadir,
- '--publisher-server', 'dbname=postgres'
],
'no subscriber connection string specified');
command_fails(
@@ -36,7 +29,6 @@ command_fails(
'pg_createsubscriber',
'--verbose',
'--pgdata', $datadir,
- '--publisher-server', 'dbname=postgres',
'--subscriber-server', 'dbname=postgres'
],
'no database name specified');
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
index 534bc53a76..a9d03acc87 100644
--- a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -56,19 +56,17 @@ command_fails(
[
'pg_createsubscriber', '--verbose',
'--pgdata', $node_f->data_dir,
- '--publisher-server', $node_p->connstr('pg1'),
'--subscriber-server', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
- 'subscriber data directory is not a copy of the source database cluster');
+ 'target database is not a physical standby');
# dry run mode on node S
command_ok(
[
'pg_createsubscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
- '--publisher-server', $node_p->connstr('pg1'),
'--subscriber-server', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
@@ -88,7 +86,6 @@ command_ok(
[
'pg_createsubscriber', '--verbose',
'--pgdata', $node_s->data_dir,
- '--publisher-server', $node_p->connstr('pg1'),
'--subscriber-server', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
--
2.43.0
[application/octet-stream] v12-0004-Exit-earlier-when-we-are-in-the-dry_run-mode.patch (11.0K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/5-v12-0004-Exit-earlier-when-we-are-in-the-dry_run-mode.patch)
download | inline diff:
From c1aa11c772925044318fc0288e5a16e6bc66f977 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 31 Jan 2024 10:45:21 +0000
Subject: [PATCH v12 4/5] Exit earlier when we are in the dry_run mode
---
src/bin/pg_basebackup/pg_createsubscriber.c | 198 ++++++++------------
1 file changed, 75 insertions(+), 123 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index f0e9db7793..6f832f7551 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -584,8 +584,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
cf->system_identifier |= getpid() & 0xFFF;
- if (!dry_run)
- update_controlfile(datadir, cf, true);
+ update_controlfile(datadir, cf, true);
pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
@@ -595,14 +594,12 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_debug("command is: %s", cmd_str);
- if (!dry_run)
- {
- rc = system(cmd_str);
- if (rc == 0)
- pg_log_info("subscriber successfully changed the system identifier");
- else
- pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
- }
+ rc = system(cmd_str);
+
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
pfree(cf);
}
@@ -676,7 +673,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
dbinfo[i].subname = pg_strdup(replslotname);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL)
pg_log_info("create replication slot \"%s\" on publisher", replslotname);
else
return false;
@@ -956,26 +953,20 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
- return lsn;
- }
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
}
/* for cleanup purposes */
if (!transient_replslot)
dbinfo->made_replslot = true;
- if (!dry_run)
- {
- lsn = pg_strdup(PQgetvalue(res, 0, 1));
- PQclear(res);
- }
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
destroyPQExpBuffer(str);
@@ -997,15 +988,12 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
- {
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQerrorMessage(conn));
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
- PQclear(res);
- }
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1138,11 +1126,8 @@ wait_for_end_recovery(const char *conninfo)
PQclear(res);
- /*
- * Does the recovery process finish? In dry run mode, there is no
- * recovery mode. Bail out as the recovery process has ended.
- */
- if (!in_recovery || dry_run)
+ /* Does the recovery process finish? */
+ if (!in_recovery)
{
status = POSTMASTER_READY;
break;
@@ -1239,24 +1224,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- {
- pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
- PQfinish(conn);
- exit(1);
- }
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
}
/* for cleanup purposes */
dbinfo->made_publication = true;
- if (!dry_run)
- PQclear(res);
-
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1277,14 +1257,11 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
- {
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
- PQclear(res);
- }
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1318,24 +1295,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- {
- pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
- PQfinish(conn);
- exit(1);
- }
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
}
/* for cleanup purposes */
dbinfo->made_subscription = true;
- if (!dry_run)
- PQclear(res);
-
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1356,14 +1328,11 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
- {
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
- PQclear(res);
- }
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1402,7 +1371,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
exit(1);
}
- if (PQntuples(res) != 1 && !dry_run)
+ if (PQntuples(res) != 1)
{
pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
PQntuples(res), 1);
@@ -1411,16 +1380,8 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
exit(1);
}
- if (dry_run)
- {
- suboid = InvalidOid;
- snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
- }
- else
- {
- suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
- snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
- }
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
/*
* The origin name is defined as pg_%u. %u is the subscription OID. See
@@ -1439,20 +1400,16 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
- PQfinish(conn);
- exit(1);
- }
-
- PQclear(res);
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
}
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1477,20 +1434,16 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- {
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
- PQerrorMessage(conn));
- PQfinish(conn);
- exit(1);
- }
-
- PQclear(res);
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
}
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1759,6 +1712,14 @@ main(int argc, char **argv)
exit(1);
}
+ /*
+ * Exit earlier when we are in the dry_run mode.
+ *
+ * XXX: Should we keep turning on the standby server in case of dry_run?
+ */
+ if (dry_run)
+ goto cleanup;
+
/*
* Create a temporary logical replication slot to get a consistent LSN.
*
@@ -1783,26 +1744,16 @@ main(int argc, char **argv)
* Despite of the recovery parameters will be written to the subscriber,
* use a publisher connection for the follwing recovery functions. The
* connection is only used to check the current server version (physical
- * replica, same server version). The subscriber is not running yet. In
- * dry run mode, the recovery parameters *won't* be written. An invalid
- * LSN is used for printing purposes.
+ * replica, same server version). The subscriber is not running yet.
*/
recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
- if (dry_run)
- {
- appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
- appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
- LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
- }
- else
- {
- appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
- consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
- }
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+
disconnect_database(conn);
pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
@@ -1860,6 +1811,7 @@ main(int argc, char **argv)
*/
modify_sysid(pg_resetwal_path, subscriber_dir);
+cleanup:
/*
* The log file is kept if retain option is specified or this tool does
* not run successfully. Otherwise, log file is removed.
--
2.43.0
[application/octet-stream] v12-0005-Divide-LogicalReplInfo-into-some-strcutures.patch (49.5K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/6-v12-0005-Divide-LogicalReplInfo-into-some-strcutures.patch)
download | inline diff:
From dd7be7f34e1ac68c2b6fd193dec2aa6b2832e1b9 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 07:03:59 +0000
Subject: [PATCH v12 5/5] Divide LogicalReplInfo into some strcutures
---
src/bin/pg_basebackup/pg_createsubscriber.c | 660 ++++++++++--------
.../t/041_pg_createsubscriber_standby.pl | 2 +-
src/tools/pgindent/typedefs.list | 5 +-
3 files changed, 365 insertions(+), 302 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index 6f832f7551..898fa3f114 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -32,54 +32,78 @@
#define PGS_OUTPUT_DIR "pg_createsubscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct LogicalRepPerdbInfoArr
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+ bool made_transient_replslot;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ char *server_log;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname);
-static bool get_exec_path(const char *path);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
-static char *get_primary_conninfo(const char *base_conninfo);
+static char *get_primary_conninfo(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static bool check_publisher(LogicalRepInfo *dbinfo);
-static bool setup_publisher(LogicalRepInfo *dbinfo);
-static bool check_subscriber(LogicalRepInfo *dbinfo);
-static bool setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_sysid_for_standby(StandbyInfo *standby);
+static void modify_sysid(const char *bindir, const char *datadir);
+static bool check_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr);
+static bool setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr);
+static bool check_subscriber(StandbyInfo *standby, LogicalRepPerdbInfoArr *dbarr);
+static bool setup_subscriber(StandbyInfo *standby, PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr,
+ const char *consistent_lsn);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static char *server_logfile_name(const char *datadir);
-static void start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile);
-static void stop_standby_server(const char *pg_ctl_path, const char *datadir);
+static void start_standby_server(StandbyInfo *standby);
+static void stop_standby_server(StandbyInfo *standby);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+
+static void wait_for_end_recovery(StandbyInfo *standby,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
#define USEC_PER_SEC 1000000
#define WAIT_INTERVAL 1 /* 1 second */
@@ -87,21 +111,17 @@ static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
/* Options */
static const char *progname;
-static char *subscriber_dir = NULL;
static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
static bool dry_run = false;
static bool retain = false;
static int recovery_timeout = 0;
static bool success = false;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
-
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
enum WaitPMResult
{
@@ -111,6 +131,30 @@ enum WaitPMResult
POSTMASTER_FAILED
};
+/*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 42
+ * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_createsubscriber_%u_%d", oid, pid);
+}
+
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 31 characters (20 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
+{
+ snprintf(pubname, szpub, "pg_createsubscriber_%u", oid);
+}
+
/*
* Cleanup objects that were created by pg_createsubscriber if there is an error.
@@ -128,33 +172,54 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
- disconnect_database(conn);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
}
}
}
+
+ if (primary.made_transient_replslot)
+ {
+ char transient_replslot[NAMEDATALEN];
+
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+
+ if (conn != NULL)
+ {
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+
+ drop_replication_slot(conn, &dbarr.perdb[0], transient_replslot);
+ disconnect_database(conn);
+ }
+ }
}
static void
@@ -236,15 +301,16 @@ get_base_conninfo(char *conninfo, char *dbname)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
- int rc;
+ int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -269,30 +335,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -356,45 +404,31 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases.
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
- SimpleStringListCell *cell;
- int i = 0;
+ SimpleStringListCell *cell;
+ int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ dbarr.perdb = (LogicalRepPerdbInfo *) pg_malloc0(ndbs *
+ sizeof(LogicalRepPerdbInfo));
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
+ char *conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
conn = PQconnectdb(conninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -412,6 +446,7 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(conninfo);
return conn;
}
@@ -429,11 +464,10 @@ disconnect_database(PGconn *conn)
* worker.
*/
static char *
-get_primary_conninfo(const char *base_conninfo)
+get_primary_conninfo(StandbyInfo *standby)
{
PGconn *conn;
PGresult *res;
- char *conninfo;
char *primaryconninfo;
pg_log_info("getting primary_conninfo from standby");
@@ -443,9 +477,7 @@ get_primary_conninfo(const char *base_conninfo)
* not stored infomation yet, we must directly get the first element of the
* database list.
*/
- conninfo = concat_conninfo_dbname(base_conninfo, database_names.head->val);
-
- conn = connect_database(conninfo);
+ conn = connect_database(standby->base_conninfo, database_names.head->val);
if (conn == NULL)
exit(1);
@@ -468,7 +500,6 @@ get_primary_conninfo(const char *base_conninfo)
exit(1);
}
- pg_free(conninfo);
PQclear(res);
disconnect_database(conn);
@@ -476,19 +507,18 @@ get_primary_conninfo(const char *base_conninfo)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -511,13 +541,12 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 2), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 2), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
@@ -525,29 +554,26 @@ get_sysid_from_conn(const char *conninfo)
* if a data directory is a clone of another one. This routine is used locally
* and avoids a connection establishment.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_sysid_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -556,7 +582,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -590,7 +616,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -609,17 +635,16 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* replication.
*/
static bool
-setup_publisher(LogicalRepInfo *dbinfo)
+setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
- for (int i = 0; i < num_dbs; i++)
+ for (int i = 0; i < dbarr->ndbs; i++)
{
- char pubname[NAMEDATALEN];
- char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -639,43 +664,20 @@ setup_publisher(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 31 characters (20 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_createsubscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
/*
* Create publication on publisher. This step should be executed
* *before* promoting the subscriber to avoid any transactions between
* consistent LSN and the new publication rows (such transactions
* wouldn't see the new publication rows resulting in an error).
*/
- create_publication(conn, &dbinfo[i]);
-
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 42
- * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
- * probability of collision. By default, subscription name is used as
- * replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_createsubscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ create_publication(conn, primary, perdb);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL)
return false;
disconnect_database(conn);
@@ -688,7 +690,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
* Is the primary server ready for logical replication?
*/
static bool
-check_publisher(LogicalRepInfo *dbinfo)
+check_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -711,7 +713,7 @@ check_publisher(LogicalRepInfo *dbinfo)
* max_replication_slots >= current + number of dbs to be converted
* max_wal_senders >= current + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -749,10 +751,11 @@ check_publisher(LogicalRepInfo *dbinfo)
* use after the transformation, hence, it will be removed at the end of
* this process.
*/
- if (primary_slot_name)
+ if (standby.primary_slot_name)
{
appendPQExpBuffer(str,
- "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'",
+ standby.primary_slot_name);
pg_log_debug("command is: %s", str->data);
@@ -767,13 +770,14 @@ check_publisher(LogicalRepInfo *dbinfo)
{
pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
PQntuples(res), 1);
- pg_free(primary_slot_name); /* it is not being used. */
- primary_slot_name = NULL;
+ pg_free(standby.primary_slot_name); /* it is not being used. */
+ standby.primary_slot_name = NULL;
return false;
}
else
{
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
}
PQclear(res);
@@ -787,17 +791,21 @@ check_publisher(LogicalRepInfo *dbinfo)
return false;
}
- if (max_repslots - cur_repslots < num_dbs)
+ if (max_repslots - cur_repslots < dbarr->ndbs)
{
- pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ pg_log_error("publisher requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ cur_repslots + dbarr->ndbs);
return false;
}
- if (max_walsenders - cur_walsenders < num_dbs)
+ if (max_walsenders - cur_walsenders < dbarr->ndbs)
{
- pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
- pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain",
+ dbarr->ndbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.",
+ cur_walsenders + dbarr->ndbs);
return false;
}
@@ -808,7 +816,7 @@ check_publisher(LogicalRepInfo *dbinfo)
* Is the standby server ready for logical replication?
*/
static bool
-check_subscriber(LogicalRepInfo *dbinfo)
+check_subscriber(StandbyInfo *standby, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -828,7 +836,7 @@ check_subscriber(LogicalRepInfo *dbinfo)
* max_logical_replication_workers >= number of dbs to be converted
* max_worker_processes >= 1 + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -845,35 +853,41 @@ check_subscriber(LogicalRepInfo *dbinfo)
max_repslots = atoi(PQgetvalue(res, 1, 0));
max_wprocs = atoi(PQgetvalue(res, 2, 0));
if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
- primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+ standby->primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
- pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+ pg_log_debug("subscriber: primary_slot_name: %s", standby->primary_slot_name);
PQclear(res);
disconnect_database(conn);
- if (max_repslots < num_dbs)
+ if (max_repslots < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_lrworkers < num_dbs)
+ if (max_lrworkers < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
- pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain",
+ dbarr->ndbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_wprocs < num_dbs + 1)
+ if (max_wprocs < dbarr->ndbs + 1)
{
- pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
- pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ pg_log_error("subscriber requires %d worker processes, but only %d remain",
+ dbarr->ndbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.",
+ dbarr->ndbs + 1);
return false;
}
@@ -885,14 +899,17 @@ check_subscriber(LogicalRepInfo *dbinfo)
* enable the subscriptions. That's the last step for logical repliation setup.
*/
static bool
-setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
+setup_subscriber(StandbyInfo *standby, PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr, const char *consistent_lsn)
{
PGconn *conn;
- for (int i = 0; i < num_dbs; i++)
+ for (int i = 0; i < dbarr->ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -901,15 +918,15 @@ setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
* available on the subscriber when the physical replica is promoted.
* Remove publications from the subscriber because it has no use.
*/
- drop_publication(conn, &dbinfo[i]);
+ drop_publication(conn, perdb);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, standby, primary, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
disconnect_database(conn);
}
@@ -925,45 +942,55 @@ setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
- snprintf(slot_name, NAMEDATALEN, "pg_createsubscriber_%d_startpoint",
+ if (temporary)
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
+ else
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name,
+ NAMEDATALEN);
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "SELECT * FROM pg_create_logical_replication_slot('%s', 'pgoutput', %s, false, false);",
- slot_name, transient_replslot ? "true" : "false");
+ slot_name, temporary ? "true" : "false");
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
+
return lsn;
}
+ pg_log_info("create replication slot \"%s\" on publisher", slot_name);
+
/* for cleanup purposes */
- if (!transient_replslot)
- dbinfo->made_replslot = true;
+ if (temporary)
+ primary.made_transient_replslot = true;
+ else
+ perdb->made_replslot = true;
+
lsn = pg_strdup(PQgetvalue(res, 0, 1));
PQclear(res);
@@ -974,14 +1001,16 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "SELECT * FROM pg_drop_replication_slot('%s');",
slot_name);
@@ -990,8 +1019,9 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQerrorMessage(conn));
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname,
+ PQerrorMessage(conn));
PQclear(res);
@@ -1026,23 +1056,25 @@ server_logfile_name(const char *datadir)
}
static void
-start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile)
+start_standby_server(StandbyInfo *standby)
{
char *pg_ctl_cmd;
int rc;
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, datadir, logfile);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -l \"%s\"",
+ standby->bindir, standby->pgdata, standby->server_log);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 1);
}
static void
-stop_standby_server(const char *pg_ctl_path, const char *datadir)
+stop_standby_server(StandbyInfo *standby)
{
char *pg_ctl_cmd;
int rc;
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, datadir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s", standby->bindir,
+ standby->pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1091,7 +1123,7 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* the recovery process. By default, it waits forever.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(StandbyInfo *standby, const char *dbname)
{
PGconn *conn;
PGresult *res;
@@ -1100,7 +1132,7 @@ wait_for_end_recovery(const char *conninfo)
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(standby->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -1139,7 +1171,7 @@ wait_for_end_recovery(const char *conninfo)
if (recovery_timeout > 0 && timer >= recovery_timeout)
{
pg_log_error("recovery timed out");
- stop_standby_server(pg_ctl_path, subscriber_dir);
+ stop_standby_server(standby);
exit(1);
}
@@ -1164,17 +1196,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -1194,7 +1230,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -1207,7 +1243,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* exact database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -1218,9 +1254,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -1228,13 +1264,13 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));;
PQfinish(conn);
exit(1);
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
PQclear(res);
destroyPQExpBuffer(str);
@@ -1244,25 +1280,29 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"",
+ pubname, perdb->dbname);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s",
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
-
destroyPQExpBuffer(str);
}
@@ -1279,19 +1319,30 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname,
+ concat_conninfo_dbname(primary->base_conninfo,
+ perdb->dbname),
+ pubname);
pg_log_debug("command is: %s", str->data);
@@ -1299,13 +1350,13 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
PQclear(res);
destroyPQExpBuffer(str);
@@ -1315,22 +1366,27 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"",
+ subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s",
+ subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
@@ -1348,18 +1404,23 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'",
+ subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1392,7 +1453,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1404,7 +1465,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1421,24 +1482,27 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
- PQerrorMessage(conn));
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
+ PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
@@ -1468,16 +1532,10 @@ main(int argc, char **argv)
int option_index;
char *base_dir;
- char *server_start_log;
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- char temp_replslot[NAMEDATALEN] = {0};
- uint64 pub_sysid;
- uint64 sub_sysid;
struct stat statbuf;
PGconn *conn;
@@ -1527,7 +1585,7 @@ main(int argc, char **argv)
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
break;
case 'S':
sub_conninfo_str = pg_strdup(optarg);
@@ -1537,7 +1595,7 @@ main(int argc, char **argv)
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
case 'n':
@@ -1573,7 +1631,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1586,8 +1644,8 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, dbname_conninfo);
- if (sub_base_conninfo == NULL)
+ standby.base_conninfo = get_base_conninfo(sub_conninfo_str, dbname_conninfo);
+ if (standby.base_conninfo == NULL)
exit(1);
if (database_names.head == NULL)
@@ -1602,7 +1660,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the subscriber connection string",
dbname_conninfo);
@@ -1616,20 +1674,20 @@ main(int argc, char **argv)
}
/* Obtain a connection string from the target */
- pub_base_conninfo = get_primary_conninfo(sub_base_conninfo);
+ primary.base_conninfo = get_primary_conninfo(&standby);
/*
* Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/* Register a function to clean up objects in case of failure. */
atexit(cleanup_objects_atexit);
@@ -1638,9 +1696,9 @@ main(int argc, char **argv)
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_sysid_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1650,7 +1708,7 @@ main(int argc, char **argv)
* Create the output directory to store any data generated by this tool.
*/
base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1663,10 +1721,10 @@ main(int argc, char **argv)
exit(1);
}
- server_start_log = server_logfile_name(subscriber_dir);
+ standby.server_log = server_logfile_name(standby.pgdata);
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* The standby server must be running. That's because some checks will be
@@ -1679,7 +1737,7 @@ main(int argc, char **argv)
/*
* Check if the standby server is ready for logical replication.
*/
- if (!check_subscriber(dbinfo))
+ if (!check_subscriber(&standby, &dbarr))
exit(1);
/*
@@ -1688,7 +1746,7 @@ main(int argc, char **argv)
* relies on check_subscriber() to obtain the primary_slot_name.
* That's why it is called after it.
*/
- if (!check_publisher(dbinfo))
+ if (!check_publisher(&primary, &dbarr))
exit(1);
/*
@@ -1697,13 +1755,13 @@ main(int argc, char **argv)
* if the primary slot is in use. We could use an extra connection for
* it but it doesn't seem worth.
*/
- if (!setup_publisher(dbinfo))
+ if (!setup_publisher(&primary, &dbarr))
exit(1);
/* Stop the standby server. */
pg_log_info("standby is up and running");
pg_log_info("stopping the server to start the transformation steps");
- stop_standby_server(pg_ctl_path, subscriber_dir);
+ stop_standby_server(&standby);
}
else
{
@@ -1732,11 +1790,10 @@ main(int argc, char **argv)
* consistent LSN but it should be changed after adding pg_basebackup
* support.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true, &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1752,7 +1809,7 @@ main(int argc, char **argv)
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
disconnect_database(conn);
@@ -1762,12 +1819,12 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
- start_standby_server(pg_ctl_path, subscriber_dir, server_start_log);
+ start_standby_server(&standby);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(&standby, dbarr.perdb[0].dbname);
/*
* Create the subscription for each database on subscriber. It does not
@@ -1776,7 +1833,7 @@ main(int argc, char **argv)
* set_replication_progress). It also cleans up publications created by
* this tool and replication to the standby.
*/
- if (!setup_subscriber(dbinfo, consistent_lsn))
+ if (!setup_subscriber(&standby, &primary, &dbarr, consistent_lsn))
exit(1);
/*
@@ -1785,12 +1842,15 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops this replication slot later.
*/
- if (primary_slot_name != NULL)
+ if (standby.primary_slot_name != NULL)
{
- conn = connect_database(dbinfo[0].pubconninfo);
+ char *primary_slot_name = standby.primary_slot_name;
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ drop_replication_slot(conn, perdb, primary_slot_name);
}
else
{
@@ -1804,12 +1864,12 @@ main(int argc, char **argv)
* Stop the subscriber.
*/
pg_log_info("stopping the subscriber");
- stop_standby_server(pg_ctl_path, subscriber_dir);
+ stop_standby_server(&standby);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
cleanup:
/*
@@ -1817,7 +1877,7 @@ cleanup:
* not run successfully. Otherwise, log file is removed.
*/
if (!retain)
- unlink(server_start_log);
+ unlink(standby.server_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
index a9d03acc87..856bf0de3c 100644
--- a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -88,7 +88,7 @@ command_ok(
'--pgdata', $node_s->data_dir,
'--subscriber-server', $node_s->connstr('pg1'),
'--database', 'pg1',
- '--database', 'pg2'
+ '--database', 'pg2', '-r'
],
'run pg_createsubscriber on node S');
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f51f1ff23f..3b1ec3fce1 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1505,9 +1505,10 @@ LogicalRepBeginData
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
-LogicalRepInfo
LogicalRepMsgType
LogicalRepPartMapEntry
+LogicalRepPerdbInfo
+LogicalRepPerdbInfoArr
LogicalRepPreparedTxnData
LogicalRepRelId
LogicalRepRelMapEntry
@@ -1886,6 +1887,7 @@ PREDICATELOCK
PREDICATELOCKTAG
PREDICATELOCKTARGET
PREDICATELOCKTARGETTAG
+PrimaryInfo
PROCESS_INFORMATION
PROCLOCK
PROCLOCKTAG
@@ -2461,6 +2463,7 @@ SQLValueFunctionOp
SSL
SSLExtensionInfoContext
SSL_CTX
+StandbyInfo
STARTUPINFO
STRLEN
SV
--
2.43.0
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-31 14:25 ` Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 14:25 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024 at 9:52 AM Hayato Kuroda (Fujitsu) <
[email protected]> wrote:
>
> Dear Euler,
>
> I extracted some review comments which may require many efforts. I hope
this makes them
> easy to review.
>
> 0001: not changed from yours.
> 0002: avoid to use replication connections. Source: comment #3[1]
> 0003: Remove -P option and use primary_conninfo instead. Source: [2]
> 0004: Exit earlier when dry_run is specified. Source: [3]
> 0005: Refactor data structures. Source: [4]
>
> [1]:
https://www.postgresql.org/message-id/TY3PR01MB9889593399165B9A04106741F5662%40TY3PR01MB9889.jpnprd0...
> [2]:
https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
> [3]:
https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
> [4]:
https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
>
Hey folks,
Jumping into this a bit late here... I'm trying a simple
pg_createsubscriber but getting an error:
~/pgsql took 19s
✦ ➜ pg_createsubscriber -d fabrizio -r -D /tmp/replica5434 -S 'host=/tmp
port=5434'
pg_createsubscriber: error: could not create subscription
"pg_createsubscriber_16384_695617" on database "fabrizio": ERROR: syntax
error at or near "/"
LINE 1: ..._16384_695617 CONNECTION 'user=fabrizio passfile='/home/fabr...
^
pg_createsubscriber: error: could not drop replication slot
"pg_createsubscriber_16384_695617" on database "fabrizio":
pg_createsubscriber: error: could not drop replication slot
"pg_subscriber_695617_startpoint" on database "fabrizio": ERROR:
replication slot "pg_subscriber_695617_startpoint" does not exist
And the LOG contains the following:
~/pgsql took 12s
✦ ➜ cat
/tmp/replica5434/pg_createsubscriber_output.d/server_start_20240131T110318.730.log
2024-01-31 11:03:19.138 -03 [695632] LOG: starting PostgreSQL 17devel on
x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0,
64-bit
2024-01-31 11:03:19.138 -03 [695632] LOG: listening on IPv6 address "::1",
port 5434
2024-01-31 11:03:19.138 -03 [695632] LOG: listening on IPv4 address
"127.0.0.1", port 5434
2024-01-31 11:03:19.158 -03 [695632] LOG: listening on Unix socket
"/tmp/.s.PGSQL.5434"
2024-01-31 11:03:19.179 -03 [695645] LOG: database system was shut down in
recovery at 2024-01-31 11:03:18 -03
2024-01-31 11:03:19.180 -03 [695645] LOG: entering standby mode
2024-01-31 11:03:19.192 -03 [695645] LOG: redo starts at 0/4000028
2024-01-31 11:03:19.198 -03 [695645] LOG: consistent recovery state
reached at 0/504DB08
2024-01-31 11:03:19.198 -03 [695645] LOG: invalid record length at
0/504DB08: expected at least 24, got 0
2024-01-31 11:03:19.198 -03 [695632] LOG: database system is ready to
accept read-only connections
2024-01-31 11:03:19.215 -03 [695646] LOG: started streaming WAL from
primary at 0/5000000 on timeline 1
2024-01-31 11:03:29.587 -03 [695645] LOG: recovery stopping after WAL
location (LSN) "0/504F260"
2024-01-31 11:03:29.587 -03 [695645] LOG: redo done at 0/504F260 system
usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 10.39 s
2024-01-31 11:03:29.587 -03 [695645] LOG: last completed transaction was
at log time 2024-01-31 11:03:18.761544-03
2024-01-31 11:03:29.587 -03 [695646] FATAL: terminating walreceiver
process due to administrator command
2024-01-31 11:03:29.598 -03 [695645] LOG: selected new timeline ID: 2
2024-01-31 11:03:29.680 -03 [695645] LOG: archive recovery complete
2024-01-31 11:03:29.690 -03 [695643] LOG: checkpoint starting:
end-of-recovery immediate wait
2024-01-31 11:03:29.795 -03 [695643] LOG: checkpoint complete: wrote 51
buffers (0.3%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.021 s,
sync=0.034 s, total=0.115 s; sync files=17, longest=0.011 s, average=0.002
s; distance=16700 kB, estimate=16700 kB; lsn=0/504F298, redo lsn=0/504F298
2024-01-31 11:03:29.805 -03 [695632] LOG: database system is ready to
accept connections
2024-01-31 11:03:30.332 -03 [695658] ERROR: syntax error at or near "/" at
character 90
2024-01-31 11:03:30.332 -03 [695658] STATEMENT: CREATE SUBSCRIPTION
pg_createsubscriber_16384_695617 CONNECTION 'user=fabrizio
passfile='/home/fabrizio/.pgpass' channel_binding=prefer host=localhost
port=5432 sslmode=prefer sslcompression=0 sslcertmode=allow sslsni=1
ssl_min_protocol_version=TLSv1.2 gssencmode=disable krbsrvname=postgres
gssdelegation=0 target_session_attrs=any load_balance_hosts=disable
dbname=fabrizio' PUBLICATION pg_createsubscriber_16384 WITH (create_slot =
false, copy_data = false, enabled = false)
Seems we need to escape connection params similar we do in dblink [1]
Regards,
[1]
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=contrib/dblink/dblink.c;h=19a362526d21d...
--
Fabrízio de Royes Mello
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
@ 2024-01-31 14:34 ` Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Euler Taveira @ 2024-01-31 14:34 UTC (permalink / raw)
To: Fabrízio de Royes Mello <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024, at 11:25 AM, Fabrízio de Royes Mello wrote:
> Jumping into this a bit late here... I'm trying a simple pg_createsubscriber but getting an error:
Try v11. It seems v12-0002 is not correct.
> Seems we need to escape connection params similar we do in dblink [1]
I think it is a consequence of v12-0003. I didn't review v12 yet but although I
have added a comment saying it might be possible to use primary_conninfo, I'm
not 100% convinced that's the right direction.
/*
* TODO use primary_conninfo (if available) from subscriber and
* extract publisher connection string. Assume that there are
* identical entries for physical and logical replication. If there is
* not, we would fail anyway.
*/
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-31 14:55 ` Fabrízio de Royes Mello <[email protected]>
2024-01-31 15:37 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 14:55 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024 at 11:35 AM Euler Taveira <[email protected]> wrote:
>
> On Wed, Jan 31, 2024, at 11:25 AM, Fabrízio de Royes Mello wrote:
>
> Jumping into this a bit late here... I'm trying a simple
pg_createsubscriber but getting an error:
>
>
> Try v11. It seems v12-0002 is not correct.
Using v11 I'm getting this error:
~/pgsql took 22s
✦ ➜ pg_createsubscriber -d fabrizio -r -D /tmp/replica5434 -S 'host=/tmp
port=5434' -P 'host=/tmp port=5432'
NOTICE: changed the failover state of replication slot
"pg_createsubscriber_16384_706609" on publisher to false
pg_createsubscriber: error: could not drop replication slot
"pg_createsubscriber_706609_startpoint" on database "fabrizio": ERROR:
replication slot "pg_createsubscriber_706609_startpoint" does not exist
Write-ahead log reset
Attached the output log.
Regards,
--
Fabrízio de Royes Mello
Attachments:
[text/x-log] server_start_20240131T115324.735.log (3.2K, ../../CAFcNs+rSG9DcEewsoA=85DXhSRh+nyKrrcr64FEDytcZf6QaEQ@mail.gmail.com/3-server_start_20240131T115324.735.log)
download | inline:
2024-01-31 11:53:31.882 -03 [706626] LOG: starting PostgreSQL 17devel on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
2024-01-31 11:53:31.882 -03 [706626] LOG: listening on IPv6 address "::1", port 5434
2024-01-31 11:53:31.882 -03 [706626] LOG: listening on IPv4 address "127.0.0.1", port 5434
2024-01-31 11:53:31.990 -03 [706626] LOG: listening on Unix socket "/tmp/.s.PGSQL.5434"
2024-01-31 11:53:32.082 -03 [706639] LOG: database system was shut down in recovery at 2024-01-31 11:53:30 -03
2024-01-31 11:53:32.084 -03 [706639] LOG: entering standby mode
2024-01-31 11:53:32.100 -03 [706639] LOG: redo starts at 0/4000028
2024-01-31 11:53:32.189 -03 [706639] LOG: consistent recovery state reached at 0/69EFF00
2024-01-31 11:53:32.189 -03 [706639] LOG: invalid record length at 0/69F01D0: expected at least 24, got 0
2024-01-31 11:53:32.189 -03 [706626] LOG: database system is ready to accept read-only connections
2024-01-31 11:53:32.200 -03 [706640] LOG: started streaming WAL from primary at 0/6000000 on timeline 1
2024-01-31 11:53:32.218 -03 [706639] LOG: recovery stopping after WAL location (LSN) "0/6A1F4C0"
2024-01-31 11:53:32.218 -03 [706639] LOG: redo done at 0/6A1F4C0 system usage: CPU: user: 0.07 s, system: 0.01 s, elapsed: 0.11 s
2024-01-31 11:53:32.219 -03 [706639] LOG: last completed transaction was at log time 2024-01-31 11:53:31.745182-03
2024-01-31 11:53:32.220 -03 [706640] FATAL: terminating walreceiver process due to administrator command
2024-01-31 11:53:32.266 -03 [706639] LOG: selected new timeline ID: 2
2024-01-31 11:53:32.390 -03 [706639] LOG: archive recovery complete
2024-01-31 11:53:32.403 -03 [706637] LOG: checkpoint starting: end-of-recovery immediate wait
2024-01-31 11:53:32.809 -03 [706637] LOG: checkpoint complete: wrote 2141 buffers (13.1%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.067 s, sync=0.198 s, total=0.419 s; sync files=51, longest=0.018 s, average=0.004 s; distance=43133 kB, estimate=43133 kB; lsn=0/6A1F4F8, redo lsn=0/6A1F4F8
2024-01-31 11:53:32.827 -03 [706626] LOG: database system is ready to accept connections
2024-01-31 11:53:33.398 -03 [706626] LOG: received fast shutdown request
2024-01-31 11:53:33.409 -03 [706650] LOG: logical replication apply worker for subscription "pg_createsubscriber_16384_706609" has started
2024-01-31 11:53:33.414 -03 [706626] LOG: aborting any active transactions
2024-01-31 11:53:33.414 -03 [706650] FATAL: terminating logical replication worker due to administrator command
2024-01-31 11:53:33.415 -03 [706626] LOG: background worker "logical replication launcher" (PID 706645) exited with exit code 1
2024-01-31 11:53:33.415 -03 [706626] LOG: background worker "logical replication apply worker" (PID 706650) exited with exit code 1
2024-01-31 11:53:33.415 -03 [706637] LOG: shutting down
2024-01-31 11:53:33.438 -03 [706637] LOG: checkpoint starting: shutdown immediate
2024-01-31 11:53:33.642 -03 [706637] LOG: checkpoint complete: wrote 23 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.052 s, sync=0.047 s, total=0.228 s; sync files=14, longest=0.011 s, average=0.004 s; distance=3 kB, estimate=38820 kB; lsn=0/6A20270, redo lsn=0/6A20270
2024-01-31 11:53:33.648 -03 [706626] LOG: database system is shut down
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
@ 2024-01-31 15:37 ` Euler Taveira <[email protected]>
2024-01-31 17:39 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Euler Taveira @ 2024-01-31 15:37 UTC (permalink / raw)
To: Fabrízio de Royes Mello <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024, at 11:55 AM, Fabrízio de Royes Mello wrote:
>
> On Wed, Jan 31, 2024 at 11:35 AM Euler Taveira <[email protected]> wrote:
> >
> > On Wed, Jan 31, 2024, at 11:25 AM, Fabrízio de Royes Mello wrote:
> >
> > Jumping into this a bit late here... I'm trying a simple pg_createsubscriber but getting an error:
> >
> >
> > Try v11. It seems v12-0002 is not correct.
>
> Using v11 I'm getting this error:
>
> ~/pgsql took 22s
> ✦ ➜ pg_createsubscriber -d fabrizio -r -D /tmp/replica5434 -S 'host=/tmp port=5434' -P 'host=/tmp port=5432'
> NOTICE: changed the failover state of replication slot "pg_createsubscriber_16384_706609" on publisher to false
> pg_createsubscriber: error: could not drop replication slot "pg_createsubscriber_706609_startpoint" on database "fabrizio": ERROR: replication slot "pg_createsubscriber_706609_startpoint" does not exist
> Write-ahead log reset
Hmm. I didn't try it with the failover patch that was recently applied. Did you
have any special configuration on primary?
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 15:37 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-31 17:39 ` Fabrízio de Royes Mello <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 17:39 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024 at 12:38 PM Euler Taveira <[email protected]> wrote:
>
>
> Hmm. I didn't try it with the failover patch that was recently applied.
Did you
> have any special configuration on primary?
>
Nothing special, here the configurations I've changed after bootstrap:
port = '5432'
wal_level = 'logical'
max_wal_senders = '8'
max_replication_slots = '6'
hot_standby_feedback = 'on'
max_prepared_transactions = '10'
max_locks_per_transaction = '512'
Regards,
--
Fabrízio de Royes Mello
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-23 23:58 ` Euler Taveira <[email protected]>
1 sibling, 0 replies; 87+ messages in thread
From: Euler Taveira @ 2024-01-23 23:58 UTC (permalink / raw)
To: [email protected] <[email protected]>; '[email protected]' <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Mon, Jan 22, 2024, at 4:06 AM, Hayato Kuroda (Fujitsu) wrote:
> I analyzed and found a reason. This is because publications are invisible for some transactions.
>
> As the first place, below operations were executed in this case.
> Tuples were inserted after getting consistent_lsn, but before starting the standby.
> After doing the workload, I confirmed again that the publication was created.
>
> 1. on primary, logical replication slots were created.
> 2. on primary, another replication slot was created.
> 3. ===on primary, some tuples were inserted. ===
> 4. on standby, a server process was started
> 5. on standby, the process waited until all changes have come.
> 6. on primary, publications were created.
> 7. on standby, subscriptions were created.
> 8. on standby, a replication progress for each subscriptions was set to given LSN (got at step2).
> =====pg_subscriber finished here=====
> 9. on standby, a server process was started again
> 10. on standby, subscriptions were enabled. They referred slots created at step1.
> 11. on primary, decoding was started but ERROR was raised.
Good catch! It is a design flaw.
> In this case, tuples were inserted *before creating publication*.
> So I thought that the decoded transaction could not see the publication because
> it was committed after insertions.
>
> One solution is to create a publication before creating a consistent slot.
> Changes which came before creating the slot were surely replicated to the standby,
> so upcoming transactions can see the object. We are planning to patch set to fix
> the issue in this approach.
I'll include a similar code in the next patch and also explain why we should
create the publication earlier. (I'm renaming
create_all_logical_replication_slots to setup_publisher and calling
create_publication from there and also adding the proposed GUC checks in it.)
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
@ 2024-10-14 11:03 Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2024-10-14 11:03 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
> > typedef struct ReturningOption
> > {
> > NodeTag type;
> > bool isNew;
> > char *name;
> > int location;
> > } ReturningOption;
Thinking about that struct some more, I think "isNew" is better done
as an enum, since this is meant to be a generic option. So even though
it might never have more than 2 possible values, I think it's neater
done that way.
Regards,
Dean
Attachments:
[text/x-patch] support-returning-old-new-v20.patch (240.2K, ../../CAEZATCXK4zsURA7aMjbhFVd+mxo8XKm+bF1JCv6RZH7fxkXtgQ@mail.gmail.com/2-support-returning-old-new-v20.patch)
download | inline diff:
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
new file mode 100644
index f2bcd6a..701e6b5
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -4975,12 +4975,12 @@ INSERT INTO ft2 (c1,c2,c3) SELECT c1+100
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
- c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
-------+-----+-----+----+----+----+------------+----
- 1101 | 201 | aaa | | | | ft2 |
- 1102 | 202 | bbb | | | | ft2 |
- 1103 | 203 | ccc | | | | ft2 |
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
+ old | new | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+-----+---------------------------------+----+----+----+----+----+----+----+----+------+-----+-----+----+----+----+------------+----
+ | (1101,201,aaa,,,,"ft2 ",) | | | | | | | | | 1101 | 201 | aaa | | | | ft2 |
+ | (1102,202,bbb,,,,"ft2 ",) | | | | | | | | | 1102 | 202 | bbb | | | | ft2 |
+ | (1103,203,ccc,,,,"ft2 ",) | | | | | | | | | 1103 | 203 | ccc | | | | ft2 |
(3 rows)
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
@@ -5111,6 +5111,31 @@ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 ||
1017 | 507 | 0001700017_update7 | | | | ft2 |
(102 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.c1, old.c2, old.c3, old.c4, old.c5, old.c6, old.c7, old.c8, new.c1, new.c2, new.c3, new.c4, new.c5, new.c6, new.c7, new.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c2 = $2, c3 = $3 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: (c2 + 400), (c3 || '_update7b'::text), ctid, ft2.*
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 7)) FOR UPDATE
+(6 rows)
+
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+----+-----+---------------+------------------------------+--------------------------+----+------------+-----+----+-----+------------------------+------------------------------+--------------------------+----+------------+-----
+ 7 | 407 | 00007_update7 | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo | 7 | 807 | 00007_update7_update7b | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo
+ 17 | 407 | 00017_update7 | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo | 17 | 807 | 00017_update7_update7b | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo
+ 27 | 407 | 00027_update7 | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo | 27 | 807 | 00027_update7_update7b | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo
+ 37 | 407 | 00037_update7 | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo | 37 | 807 | 00037_update7_update7b | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -5241,6 +5266,29 @@ DELETE FROM ft2 WHERE c1 % 10 = 5 RETURN
1105 |
(103 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------
+ Delete on public.ft2
+ Output: old.c1, c4
+ Remote SQL: DELETE FROM "S 1"."T 1" WHERE ctid = $1 RETURNING "C 1", c4
+ -> Foreign Scan on public.ft2
+ Output: ctid
+ Remote SQL: SELECT ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 6)) FOR UPDATE
+(6 rows)
+
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ c1 | c4
+----+------------------------------
+ 6 | Wed Jan 07 00:00:00 1970 PST
+ 16 | Sat Jan 17 00:00:00 1970 PST
+ 26 | Tue Jan 27 00:00:00 1970 PST
+ 36 | Fri Feb 06 00:00:00 1970 PST
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
QUERY PLAN
@@ -6165,6 +6213,70 @@ UPDATE ft2 SET c3 = 'foo'
(1296,96,foo,,,,"ft2 ",) | 1296 | 96 | foo | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
(16 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.*, new.*, ft2.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan
+ Output: 'bar'::text, ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Relations: ((public.ft2) INNER JOIN (public.ft4)) INNER JOIN (public.ft5)
+ Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2.c1, r2.c2, r2.c3) END, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r2.c1, r2.c2, r2.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r2 ON (((r1.c2 = r2.c1)) AND ((r1."C 1" > 1200)))) INNER JOIN "S 1"."T 4" r3 ON (((r2.c1 = r3.c1)))) FOR UPDATE OF r1
+ -> Nested Loop
+ Output: ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Join Filter: (ft4.c1 = ft5.c1)
+ -> Sort
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Sort Key: ft2.c2
+ -> Hash Join
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Hash Cond: (ft2.c2 = ft4.c1)
+ -> Foreign Scan on public.ft2
+ Output: ft2.ctid, ft2.*, ft2.c2
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" > 1200)) FOR UPDATE
+ -> Hash
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ -> Foreign Scan on public.ft4
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 3"
+ -> Materialize
+ Output: ft5.*, ft5.c1
+ -> Foreign Scan on public.ft5
+ Output: ft5.*, ft5.c1
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 4"
+(29 rows)
+
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ old | new | ft2 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | ft4 | c1 | c2 | c3
+--------------------------------+--------------------------------+--------------------------------+------+----+-----+----+----+----+------------+----+----------------+----+----+--------
+ (1206,6,foo,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | 1206 | 6 | bar | | | | ft2 | | (6,7,AAA006) | 6 | 7 | AAA006
+ (1212,12,foo,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | 1212 | 12 | bar | | | | ft2 | | (12,13,AAA012) | 12 | 13 | AAA012
+ (1224,24,foo,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | 1224 | 24 | bar | | | | ft2 | | (24,25,AAA024) | 24 | 25 | AAA024
+ (1230,30,foo,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | 1230 | 30 | bar | | | | ft2 | | (30,31,AAA030) | 30 | 31 | AAA030
+ (1242,42,foo,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | 1242 | 42 | bar | | | | ft2 | | (42,43,AAA042) | 42 | 43 | AAA042
+ (1248,48,foo,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | 1248 | 48 | bar | | | | ft2 | | (48,49,AAA048) | 48 | 49 | AAA048
+ (1260,60,foo,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | 1260 | 60 | bar | | | | ft2 | | (60,61,AAA060) | 60 | 61 | AAA060
+ (1266,66,foo,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | 1266 | 66 | bar | | | | ft2 | | (66,67,AAA066) | 66 | 67 | AAA066
+ (1278,78,foo,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | 1278 | 78 | bar | | | | ft2 | | (78,79,AAA078) | 78 | 79 | AAA078
+ (1284,84,foo,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | 1284 | 84 | bar | | | | ft2 | | (84,85,AAA084) | 84 | 85 | AAA084
+ (1296,96,foo,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | 1296 | 96 | bar | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
+ (1218,18,foo,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | 1218 | 18 | bar | | | | ft2 | | (18,19,AAA018) | 18 | 19 | AAA018
+ (1236,36,foo,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | 1236 | 36 | bar | | | | ft2 | | (36,37,AAA036) | 36 | 37 | AAA036
+ (1254,54,foo,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | 1254 | 54 | bar | | | | ft2 | | (54,55,AAA054) | 54 | 55 | AAA054
+ (1272,72,foo,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | 1272 | 72 | bar | | | | ft2 | | (72,73,AAA072) | 72 | 73 | AAA072
+ (1290,90,foo,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | 1290 | 90 | bar | | | | ft2 | | (90,91,AAA090) | 90 | 91 | AAA090
+(16 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
new file mode 100644
index 372fe6d..c704dae
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1469,7 +1469,7 @@ EXPLAIN (verbose, costs off)
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 300, c3 = c3 || '_update3' WHERE c1 % 10 = 3; -- can be pushed down
@@ -1477,6 +1477,13 @@ UPDATE ft2 SET c2 = c2 + 300, c3 = c3 ||
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *; -- can be pushed down
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -1485,6 +1492,11 @@ UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = f
EXPLAIN (verbose, costs off)
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4; -- can be pushed down
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2;
@@ -1511,6 +1523,17 @@ UPDATE ft2 SET c3 = 'foo'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
RETURNING ft2, ft2.*, ft4, ft4.*;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/doc/src/sgml/dml.sgml b/doc/src/sgml/dml.sgml
new file mode 100644
index 3d95bdb..458aee7
--- a/doc/src/sgml/dml.sgml
+++ b/doc/src/sgml/dml.sgml
@@ -308,7 +308,8 @@ DELETE FROM products;
</para>
<para>
- In an <command>INSERT</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>INSERT</command>, the default data available to
+ <literal>RETURNING</literal> is
the row as it was inserted. This is not so useful in trivial inserts,
since it would just repeat the data provided by the client. But it can
be very handy when relying on computed default values. For example,
@@ -325,7 +326,8 @@ INSERT INTO users (firstname, lastname)
</para>
<para>
- In an <command>UPDATE</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>UPDATE</command>, the default data available to
+ <literal>RETURNING</literal> is
the new content of the modified row. For example:
<programlisting>
UPDATE products SET price = price * 1.10
@@ -335,7 +337,8 @@ UPDATE products SET price = price * 1.10
</para>
<para>
- In a <command>DELETE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>DELETE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the deleted row. For example:
<programlisting>
DELETE FROM products
@@ -345,7 +348,8 @@ DELETE FROM products
</para>
<para>
- In a <command>MERGE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>MERGE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the source row plus the content of the inserted, updated, or
deleted target row. Since it is quite common for the source and target to
have many of the same columns, specifying <literal>RETURNING *</literal>
@@ -360,6 +364,35 @@ MERGE INTO products p USING new_products
</para>
<para>
+ In each of these commands, it is also possible to explicitly return the
+ old and new content of the modified row. For example:
+<programlisting>
+UPDATE products SET price = price * 1.10
+ WHERE price <= 99.99
+ RETURNING name, old.price AS old_price, new.price AS new_price,
+ new.price - old.price AS price_change;
+</programlisting>
+ In this example, writing <literal>new.price</literal> is the same as
+ just writing <literal>price</literal>, but it makes the meaning clearer.
+ </para>
+
+ <para>
+ This syntax for returning old and new values is available in
+ <command>INSERT</command>, <command>UPDATE</command>,
+ <command>DELETE</command>, and <command>MERGE</command> commands, but
+ typically old values will be <literal>NULL</literal> for an
+ <command>INSERT</command>, and new values will be <literal>NULL</literal>
+ for a <command>DELETE</command>. However, there are situations where it
+ can still be useful for those commands. For example, in an
+ <command>INSERT</command> with an
+ <link linkend="sql-on-conflict"><literal>ON CONFLICT DO UPDATE</literal></link>
+ clause, the old values will be non-<literal>NULL</literal> for conflicting
+ rows. Similarly, if a <command>DELETE</command> is turned into an
+ <command>UPDATE</command> by a <link linkend="sql-createrule">rewrite rule</link>,
+ the new values may be non-<literal>NULL</literal>.
+ </para>
+
+ <para>
If there are triggers (<xref linkend="triggers"/>) on the target table,
the data available to <literal>RETURNING</literal> is the row as modified by
the triggers. Thus, inspecting columns computed by triggers is another
diff --git a/doc/src/sgml/ref/delete.sgml b/doc/src/sgml/ref/delete.sgml
new file mode 100644
index 7717855..29649f6
--- a/doc/src/sgml/ref/delete.sgml
+++ b/doc/src/sgml/ref/delete.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
[ USING <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -161,6 +162,26 @@ DELETE FROM [ ONLY ] <replaceable class=
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -170,6 +191,23 @@ DELETE FROM [ ONLY ] <replaceable class=
or table(s) listed in <literal>USING</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return old values.
+ </para>
+
+ <para>
+ For a simple <command>DELETE</command>, all new values will be
+ <literal>NULL</literal>. However, if an <literal>ON DELETE</literal>
+ rule causes an <command>INSERT</command> or <command>UPDATE</command>
+ to be executed instead, the new values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml
new file mode 100644
index 6f0adee..3f13991
--- a/doc/src/sgml/ref/insert.sgml
+++ b/doc/src/sgml/ref/insert.sgml
@@ -26,7 +26,8 @@ INSERT INTO <replaceable class="paramete
[ OVERRIDING { SYSTEM | USER } VALUE ]
{ DEFAULT VALUES | VALUES ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="parameter">query</replaceable> }
[ ON CONFLICT [ <replaceable class="parameter">conflict_target</replaceable> ] <replaceable class="parameter">conflict_action</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">conflict_target</replaceable> can be one of:</phrase>
@@ -294,6 +295,26 @@ INSERT INTO <replaceable class="paramete
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -305,6 +326,23 @@ INSERT INTO <replaceable class="paramete
<literal>*</literal> to return all columns of the inserted or updated
row(s).
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values.
+ </para>
+
+ <para>
+ For a simple <command>INSERT</command>, all old values will be
+ <literal>NULL</literal>. However, for an <command>INSERT</command>
+ with an <literal>ON CONFLICT DO UPDATE</literal> clause, the old
+ values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
@@ -714,6 +752,20 @@ INSERT INTO distributors (did, dname)
</programlisting>
</para>
<para>
+ Insert or update new distributors as above, returning information
+ about any existing values that were updated, together with the new data
+ inserted. Note that the returned values for <literal>old_did</literal>
+ and <literal>old_dname</literal> will be <literal>NULL</literal> for
+ non-conflicting rows:
+<programlisting>
+INSERT INTO distributors (did, dname)
+ VALUES (5, 'Gizmo Transglobal'), (6, 'Associated Computing, Inc')
+ ON CONFLICT (did) DO UPDATE SET dname = EXCLUDED.dname
+ RETURNING old.did AS old_did, old.dname AS old_dname,
+ new.did AS new_did, new.dname AS new_dname;
+</programlisting>
+ </para>
+ <para>
Insert a distributor, or do nothing for rows proposed for insertion
when an existing, excluded row (a row with a matching constrained
column or columns after before row insert triggers fire) exists.
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
new file mode 100644
index 97b34b9..1b47e9a
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
MERGE INTO [ ONLY ] <replaceable class="parameter">target_table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">target_alias</replaceable> ]
USING <replaceable class="parameter">data_source</replaceable> ON <replaceable class="parameter">join_condition</replaceable>
<replaceable class="parameter">when_clause</replaceable> [...]
-[ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+[ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">data_source</replaceable> is:</phrase>
@@ -500,6 +501,25 @@ DELETE
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -517,6 +537,17 @@ DELETE
qualifying the <literal>*</literal> with the name or alias of the source
or target table.
</para>
+ <para>
+ A column name or <literal>*</literal> may also be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values from the target table to be returned. An unqualified column
+ name, or <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values
+ for <literal>INSERT</literal> and <literal>UPDATE</literal> actions, and
+ old values for <literal>DELETE</literal> actions.
+ </para>
</listitem>
</varlistentry>
@@ -739,7 +770,7 @@ WHEN MATCHED AND w.stock + s.stock_delta
UPDATE SET stock = w.stock + s.stock_delta
WHEN MATCHED THEN
DELETE
-RETURNING merge_action(), w.*;
+RETURNING merge_action(), w.winename, old.stock AS old_stock, new.stock AS new_stock;
</programlisting>
The <literal>wine_stock_changes</literal> table might be, for example, a
diff --git a/doc/src/sgml/ref/update.sgml b/doc/src/sgml/ref/update.sgml
new file mode 100644
index 1c433be..12ec5ba
--- a/doc/src/sgml/ref/update.sgml
+++ b/doc/src/sgml/ref/update.sgml
@@ -29,7 +29,8 @@ UPDATE [ ONLY ] <replaceable class="para
} [, ...]
[ FROM <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -212,6 +213,26 @@ UPDATE [ ONLY ] <replaceable class="para
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -221,6 +242,16 @@ UPDATE [ ONLY ] <replaceable class="para
or table(s) listed in <literal>FROM</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return new values.
+ </para>
</listitem>
</varlistentry>
@@ -348,12 +379,13 @@ UPDATE weather SET temp_lo = temp_lo+1,
</para>
<para>
- Perform the same operation and return the updated entries:
+ Perform the same operation and return the updated entries, and the old
+ precipitation value:
<programlisting>
UPDATE weather SET temp_lo = temp_lo+1, temp_hi = temp_lo+15, prcp = DEFAULT
WHERE city = 'San Francisco' AND date = '2003-07-03'
- RETURNING temp_lo, temp_hi, prcp;
+ RETURNING temp_lo, temp_hi, prcp, old.prcp AS old_prcp;
</programlisting>
</para>
diff --git a/doc/src/sgml/rules.sgml b/doc/src/sgml/rules.sgml
new file mode 100644
index 7a928bd..e992baa
--- a/doc/src/sgml/rules.sgml
+++ b/doc/src/sgml/rules.sgml
@@ -1646,6 +1646,23 @@ CREATE RULE shoelace_ins AS ON INSERT TO
</para>
<para>
+ Note that in the <literal>RETURNING</literal> clause of a rule,
+ <literal>OLD</literal> and <literal>NEW</literal> refer to the
+ pseudorelations added as extra range table entries to the rewritten
+ query, rather than old/new rows in the result relation. Thus, for
+ example, in a rule supporting <command>UPDATE</command> queries on this
+ view, if the <literal>RETURNING</literal> clause contained
+ <literal>old.sl_name</literal>, the old name would always be returned,
+ regardless of whether the <literal>RETURNING</literal> clause in the
+ query on the view specified <literal>OLD</literal> or <literal>NEW</literal>,
+ which might be confusing. To avoid this confusion, and support returning
+ old and new values in queries on the view, the <literal>RETURNING</literal>
+ clause in the rule definition should refer to entries from the result
+ relation such as <literal>shoelace_data.sl_name</literal>, without
+ specifying <literal>OLD</literal> or <literal>NEW</literal>.
+ </para>
+
+ <para>
Now assume that once in a while, a pack of shoelaces arrives at
the shop and a big parts list along with it. But you don't want
to manually update the <literal>shoelace</literal> view every
diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c
new file mode 100644
index c8077aa..8ea955f
--- a/src/backend/executor/execExpr.c
+++ b/src/backend/executor/execExpr.c
@@ -55,10 +55,15 @@
typedef struct ExprSetupInfo
{
- /* Highest attribute numbers fetched from inner/outer/scan tuple slots: */
+ /*
+ * Highest attribute numbers fetched from inner/outer/scan/old/new tuple
+ * slots:
+ */
AttrNumber last_inner;
AttrNumber last_outer;
AttrNumber last_scan;
+ AttrNumber last_old;
+ AttrNumber last_new;
/* MULTIEXPR SubPlan nodes appearing in the expression: */
List *multiexpr_subplans;
} ExprSetupInfo;
@@ -446,8 +451,25 @@ ExecBuildProjectionInfo(List *targetList
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+
+ /*
+ * Get the tuple from the relation being scanned, and keep
+ * track of whether any OLD/NEW values were requested.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_ASSIGN_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_ASSIGN_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ default:
+ scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+ break;
+ }
break;
}
@@ -535,7 +557,7 @@ ExecBuildUpdateProjection(List *targetLi
int nAssignableCols;
bool sawJunk;
Bitmapset *assignedCols;
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
ExprEvalStep scratch = {0};
int outerattnum;
ListCell *lc,
@@ -924,6 +946,7 @@ ExecInitExprRec(Expr *node, ExprState *s
/* system column */
scratch.d.var.attnum = variable->varattno;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -936,7 +959,20 @@ ExecInitExprRec(Expr *node, ExprState *s
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_SYSVAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ default:
+ scratch.opcode = EEOP_SCAN_SYSVAR;
+ break;
+ }
break;
}
}
@@ -945,6 +981,7 @@ ExecInitExprRec(Expr *node, ExprState *s
/* regular user column */
scratch.d.var.attnum = variable->varattno - 1;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -957,7 +994,20 @@ ExecInitExprRec(Expr *node, ExprState *s
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_VAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ default:
+ scratch.opcode = EEOP_SCAN_VAR;
+ break;
+ }
break;
}
}
@@ -2565,6 +2615,28 @@ ExecInitExprRec(Expr *node, ExprState *s
break;
}
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ int retstep;
+
+ /* Skip expression evaluation if OLD/NEW row doesn't exist */
+ scratch.opcode = EEOP_RETURNINGEXPR;
+ scratch.d.returningexpr.nullflag = rexpr->retold ?
+ EEO_FLAG_OLD_IS_NULL : EEO_FLAG_NEW_IS_NULL;
+ scratch.d.returningexpr.jumpdone = -1; /* set below */
+ ExprEvalPushStep(state, &scratch);
+ retstep = state->steps_len - 1;
+
+ /* Steps to evaluate expression to return */
+ ExecInitExprRec(rexpr->retexpr, state, resv, resnull);
+
+ /* Jump target used if OLD/NEW row doesn't exist */
+ state->steps[retstep].d.returningexpr.jumpdone = state->steps_len;
+
+ break;
+ }
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(node));
@@ -2776,7 +2848,7 @@ ExecInitSubPlanExpr(SubPlan *subplan,
static void
ExecCreateExprSetupSteps(ExprState *state, Node *node)
{
- ExprSetupInfo info = {0, 0, 0, NIL};
+ ExprSetupInfo info = {0, 0, 0, 0, 0, NIL};
/* Prescan to find out what we need. */
expr_setup_walker(node, &info);
@@ -2799,8 +2871,8 @@ ExecPushExprSetupSteps(ExprState *state,
scratch.resnull = NULL;
/*
- * Add steps deforming the ExprState's inner/outer/scan slots as much as
- * required by any Vars appearing in the expression.
+ * Add steps deforming the ExprState's inner/outer/scan/old/new slots as
+ * much as required by any Vars appearing in the expression.
*/
if (info->last_inner > 0)
{
@@ -2832,6 +2904,26 @@ ExecPushExprSetupSteps(ExprState *state,
if (ExecComputeSlotInfo(state, &scratch))
ExprEvalPushStep(state, &scratch);
}
+ if (info->last_old > 0)
+ {
+ scratch.opcode = EEOP_OLD_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_old;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
+ if (info->last_new > 0)
+ {
+ scratch.opcode = EEOP_NEW_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_new;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
/*
* Add steps to execute any MULTIEXPR SubPlans appearing in the
@@ -2878,7 +2970,18 @@ expr_setup_walker(Node *node, ExprSetupI
/* INDEX_VAR is handled by default case */
default:
- info->last_scan = Max(info->last_scan, attnum);
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ info->last_old = Max(info->last_old, attnum);
+ break;
+ case VAR_RETURNING_NEW:
+ info->last_new = Max(info->last_new, attnum);
+ break;
+ default:
+ info->last_scan = Max(info->last_scan, attnum);
+ break;
+ }
break;
}
return false;
@@ -2917,6 +3020,11 @@ expr_setup_walker(Node *node, ExprSetupI
* evaluation of the expression will have the same type of slot, with an
* equivalent descriptor.
*
+ * EEOP_OLD_FETCHSOME and EEOP_NEW_FETCHSOME are used to process RETURNING, if
+ * OLD/NEW columns are referred to explicitly. In both cases, the tuple
+ * descriptor comes from the parent scan node, so we treat them the same as
+ * EEOP_SCAN_FETCHSOME.
+ *
* Returns true if the deforming step is required, false otherwise.
*/
static bool
@@ -2930,7 +3038,9 @@ ExecComputeSlotInfo(ExprState *state, Ex
Assert(opcode == EEOP_INNER_FETCHSOME ||
opcode == EEOP_OUTER_FETCHSOME ||
- opcode == EEOP_SCAN_FETCHSOME);
+ opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME);
if (op->d.fetch.known_desc != NULL)
{
@@ -2982,7 +3092,9 @@ ExecComputeSlotInfo(ExprState *state, Ex
desc = ExecGetResultType(os);
}
}
- else if (opcode == EEOP_SCAN_FETCHSOME)
+ else if (opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME)
{
desc = parent->scandesc;
@@ -3030,6 +3142,12 @@ ExecInitWholeRowVar(ExprEvalStep *scratc
scratch->d.wholerow.tupdesc = NULL; /* filled at runtime */
scratch->d.wholerow.junkFilter = NULL;
+ /* update ExprState flags if Var refers to OLD/NEW */
+ if (variable->varreturningtype == VAR_RETURNING_OLD)
+ state->flags |= EEO_FLAG_HAS_OLD;
+ else if (variable->varreturningtype == VAR_RETURNING_NEW)
+ state->flags |= EEO_FLAG_HAS_NEW;
+
/*
* If the input tuple came from a subquery, it might contain "resjunk"
* columns (such as GROUP BY or ORDER BY columns), which we don't want to
@@ -3532,7 +3650,7 @@ ExecBuildAggTrans(AggState *aggstate, Ag
PlanState *parent = &aggstate->ss.ps;
ExprEvalStep scratch = {0};
bool isCombine = DO_AGGSPLIT_COMBINE(aggstate->aggsplit);
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
state->expr = (Expr *) aggstate;
state->parent = parent;
@@ -4211,6 +4329,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = latt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4219,6 +4338,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = ratt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4345,6 +4465,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4353,6 +4474,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
new file mode 100644
index 9fd988c..691e946
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -157,17 +157,25 @@ static void ExecEvalRowNullInt(ExprState
static Datum ExecJustInnerVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustOuterVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustScanVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustOldVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustNewVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignInnerVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignOuterVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignScanVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignOldVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignNewVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustApplyFuncToCase(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustConst(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustInnerVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustOuterVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustScanVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignInnerVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignOuterVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignScanVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
/* execution helper functions */
static pg_attribute_always_inline void ExecAggPlainTransByVal(AggState *aggstate,
@@ -296,6 +304,18 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustScanVar;
return;
}
+ else if (step0 == EEOP_OLD_FETCHSOME &&
+ step1 == EEOP_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustOldVar;
+ return;
+ }
+ else if (step0 == EEOP_NEW_FETCHSOME &&
+ step1 == EEOP_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustNewVar;
+ return;
+ }
else if (step0 == EEOP_INNER_FETCHSOME &&
step1 == EEOP_ASSIGN_INNER_VAR)
{
@@ -314,6 +334,18 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustAssignScanVar;
return;
}
+ else if (step0 == EEOP_OLD_FETCHSOME &&
+ step1 == EEOP_ASSIGN_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignOldVar;
+ return;
+ }
+ else if (step0 == EEOP_NEW_FETCHSOME &&
+ step1 == EEOP_ASSIGN_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignNewVar;
+ return;
+ }
else if (step0 == EEOP_CASE_TESTVAL &&
step1 == EEOP_FUNCEXPR_STRICT &&
state->steps[0].d.casetest.value)
@@ -346,6 +378,16 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustScanVarVirt;
return;
}
+ else if (step0 == EEOP_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustOldVarVirt;
+ return;
+ }
+ else if (step0 == EEOP_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustNewVarVirt;
+ return;
+ }
else if (step0 == EEOP_ASSIGN_INNER_VAR)
{
state->evalfunc_private = (void *) ExecJustAssignInnerVarVirt;
@@ -361,6 +403,16 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustAssignScanVarVirt;
return;
}
+ else if (step0 == EEOP_ASSIGN_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignOldVarVirt;
+ return;
+ }
+ else if (step0 == EEOP_ASSIGN_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignNewVarVirt;
+ return;
+ }
}
#if defined(EEO_USE_COMPUTED_GOTO)
@@ -400,6 +452,8 @@ ExecInterpExpr(ExprState *state, ExprCon
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
/*
* This array has to be in the same order as enum ExprEvalOp.
@@ -410,16 +464,24 @@ ExecInterpExpr(ExprState *state, ExprCon
&&CASE_EEOP_INNER_FETCHSOME,
&&CASE_EEOP_OUTER_FETCHSOME,
&&CASE_EEOP_SCAN_FETCHSOME,
+ &&CASE_EEOP_OLD_FETCHSOME,
+ &&CASE_EEOP_NEW_FETCHSOME,
&&CASE_EEOP_INNER_VAR,
&&CASE_EEOP_OUTER_VAR,
&&CASE_EEOP_SCAN_VAR,
+ &&CASE_EEOP_OLD_VAR,
+ &&CASE_EEOP_NEW_VAR,
&&CASE_EEOP_INNER_SYSVAR,
&&CASE_EEOP_OUTER_SYSVAR,
&&CASE_EEOP_SCAN_SYSVAR,
+ &&CASE_EEOP_OLD_SYSVAR,
+ &&CASE_EEOP_NEW_SYSVAR,
&&CASE_EEOP_WHOLEROW,
&&CASE_EEOP_ASSIGN_INNER_VAR,
&&CASE_EEOP_ASSIGN_OUTER_VAR,
&&CASE_EEOP_ASSIGN_SCAN_VAR,
+ &&CASE_EEOP_ASSIGN_OLD_VAR,
+ &&CASE_EEOP_ASSIGN_NEW_VAR,
&&CASE_EEOP_ASSIGN_TMP,
&&CASE_EEOP_ASSIGN_TMP_MAKE_RO,
&&CASE_EEOP_CONST,
@@ -461,6 +523,7 @@ ExecInterpExpr(ExprState *state, ExprCon
&&CASE_EEOP_SQLVALUEFUNCTION,
&&CASE_EEOP_CURRENTOFEXPR,
&&CASE_EEOP_NEXTVALUEEXPR,
+ &&CASE_EEOP_RETURNINGEXPR,
&&CASE_EEOP_ARRAYEXPR,
&&CASE_EEOP_ARRAYCOERCE,
&&CASE_EEOP_ROW,
@@ -529,6 +592,8 @@ ExecInterpExpr(ExprState *state, ExprCon
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
#if defined(EEO_USE_COMPUTED_GOTO)
EEO_DISPATCH();
@@ -568,6 +633,24 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, oldslot);
+
+ slot_getsomeattrs(oldslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, newslot);
+
+ slot_getsomeattrs(newslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_VAR)
{
int attnum = op->d.var.attnum;
@@ -611,6 +694,32 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ *op->resvalue = oldslot->tts_values[attnum];
+ *op->resnull = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ *op->resvalue = newslot->tts_values[attnum];
+ *op->resnull = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_SYSVAR)
{
ExecEvalSysVar(state, op, econtext, innerslot);
@@ -629,6 +738,18 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, oldslot);
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, newslot);
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_WHOLEROW)
{
/* too complex for an inline implementation */
@@ -688,6 +809,40 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_ASSIGN_OLD_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = oldslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_ASSIGN_NEW_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = newslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ASSIGN_TMP)
{
int resultnum = op->d.assign_tmp.resultnum;
@@ -1364,6 +1519,23 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_RETURNINGEXPR)
+ {
+ /*
+ * The next op actually evaluates the expression. If the OLD/NEW
+ * row doesn't exist, skip that and return NULL.
+ */
+ if (state->flags & op->d.returningexpr.nullflag)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+
+ EEO_JUMP(op->d.returningexpr.jumpdone);
+ }
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ARRAYEXPR)
{
/* too complex for an inline implementation */
@@ -2043,10 +2215,14 @@ CheckExprStillValid(ExprState *state, Ex
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
for (int i = 0; i < state->steps_len; i++)
{
@@ -2077,6 +2253,22 @@ CheckExprStillValid(ExprState *state, Ex
CheckVarSlotCompatibility(scanslot, attnum + 1, op->d.var.vartype);
break;
}
+
+ case EEOP_OLD_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(oldslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
+
+ case EEOP_NEW_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(newslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
default:
break;
}
@@ -2251,7 +2443,7 @@ get_cached_rowtype(Oid type_id, int32 ty
* Fast-path functions, for very simple expressions
*/
-/* implementation of ExecJust(Inner|Outer|Scan)Var */
+/* implementation of ExecJust(Inner|Outer|Scan|Old|New)Var */
static pg_attribute_always_inline Datum
ExecJustVarImpl(ExprState *state, TupleTableSlot *slot, bool *isnull)
{
@@ -2289,7 +2481,21 @@ ExecJustScanVar(ExprState *state, ExprCo
return ExecJustVarImpl(state, econtext->ecxt_scantuple, isnull);
}
-/* implementation of ExecJustAssign(Inner|Outer|Scan)Var */
+/* Simple reference to OLD Var in RETURNING */
+static Datum
+ExecJustOldVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Simple reference to NEW Var in RETURNING */
+static Datum
+ExecJustNewVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
+/* implementation of ExecJustAssign(Inner|Outer|Scan|Old|New)Var */
static pg_attribute_always_inline Datum
ExecJustAssignVarImpl(ExprState *state, TupleTableSlot *inslot, bool *isnull)
{
@@ -2336,6 +2542,20 @@ ExecJustAssignScanVar(ExprState *state,
return ExecJustAssignVarImpl(state, econtext->ecxt_scantuple, isnull);
}
+/* Evaluate OLD Var and assign to appropriate column of result tuple */
+static Datum
+ExecJustAssignOldVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Evaluate NEW Var and assign to appropriate column of result tuple */
+static Datum
+ExecJustAssignNewVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
/* Evaluate CASE_TESTVAL and apply a strict function to it */
static Datum
ExecJustApplyFuncToCase(ExprState *state, ExprContext *econtext, bool *isnull)
@@ -2384,7 +2604,7 @@ ExecJustConst(ExprState *state, ExprCont
return op->d.constval.value;
}
-/* implementation of ExecJust(Inner|Outer|Scan)VarVirt */
+/* implementation of ExecJust(Inner|Outer|Scan|Old|New)VarVirt */
static pg_attribute_always_inline Datum
ExecJustVarVirtImpl(ExprState *state, TupleTableSlot *slot, bool *isnull)
{
@@ -2427,7 +2647,21 @@ ExecJustScanVarVirt(ExprState *state, Ex
return ExecJustVarVirtImpl(state, econtext->ecxt_scantuple, isnull);
}
-/* implementation of ExecJustAssign(Inner|Outer|Scan)VarVirt */
+/* Like ExecJustOldVar, optimized for virtual slots */
+static Datum
+ExecJustOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarVirtImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Like ExecJustNewVar, optimized for virtual slots */
+static Datum
+ExecJustNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarVirtImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
+/* implementation of ExecJustAssign(Inner|Outer|Scan|Old|New)VarVirt */
static pg_attribute_always_inline Datum
ExecJustAssignVarVirtImpl(ExprState *state, TupleTableSlot *inslot, bool *isnull)
{
@@ -2470,6 +2704,20 @@ ExecJustAssignScanVarVirt(ExprState *sta
return ExecJustAssignVarVirtImpl(state, econtext->ecxt_scantuple, isnull);
}
+/* Like ExecJustAssignOldVar, optimized for virtual slots */
+static Datum
+ExecJustAssignOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarVirtImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Like ExecJustAssignNewVar, optimized for virtual slots */
+static Datum
+ExecJustAssignNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarVirtImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
#if defined(EEO_USE_COMPUTED_GOTO)
/*
* Comparator used when building address->opcode lookup table for
@@ -4920,8 +5168,40 @@ ExecEvalWholeRowVar(ExprState *state, Ex
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- slot = econtext->ecxt_scantuple;
+
+ /*
+ * Get the tuple from the relation being scanned.
+ *
+ * By default, this uses the "scan" tuple slot, but a wholerow Var
+ * in the RETURNING list may explicitly refer to OLD/NEW. If the
+ * OLD/NEW row doesn't exist, we just return NULL.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ Assert(state->flags & EEO_FLAG_HAS_OLD);
+ if (state->flags & EEO_FLAG_OLD_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_oldtuple;
+ break;
+ case VAR_RETURNING_NEW:
+ Assert(state->flags & EEO_FLAG_HAS_NEW);
+ if (state->flags & EEO_FLAG_NEW_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_newtuple;
+ break;
+ default:
+ slot = econtext->ecxt_scantuple;
+ break;
+ }
break;
}
@@ -5124,6 +5404,38 @@ ExecEvalSysVar(ExprState *state, ExprEva
{
Datum d;
+ /*
+ * For OLD/NEW system attributes, check whether the OLD/NEW row exists. If
+ * it doesn't, the OLD/NEW system attribute is NULL.
+ */
+ if (op->d.var.varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ bool rowIsNull;
+
+ switch (op->d.var.varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ Assert(state->flags & EEO_FLAG_HAS_OLD);
+ rowIsNull = (state->flags & EEO_FLAG_OLD_IS_NULL) != 0;
+ break;
+ case VAR_RETURNING_NEW:
+ Assert(state->flags & EEO_FLAG_HAS_NEW);
+ rowIsNull = (state->flags & EEO_FLAG_NEW_IS_NULL) != 0;
+ break;
+ default:
+ elog(ERROR, "unrecognized varreturningtype: %d",
+ (int) op->d.var.varreturningtype);
+ rowIsNull = false; /* keep compiler quiet */
+ }
+
+ if (rowIsNull)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ }
+
/* slot_getsysattr has sufficient defenses against bad attnums */
d = slot_getsysattr(slot,
op->d.var.attnum,
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
new file mode 100644
index cc9a594..594fc97
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -1255,6 +1255,7 @@ InitResultRelInfo(ResultRelInfo *resultR
resultRelInfo->ri_ReturningSlot = NULL;
resultRelInfo->ri_TrigOldSlot = NULL;
resultRelInfo->ri_TrigNewSlot = NULL;
+ resultRelInfo->ri_AllNullSlot = NULL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_MATCHED] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_SOURCE] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_TARGET] = NIL;
diff --git a/src/backend/executor/execUtils.c b/src/backend/executor/execUtils.c
new file mode 100644
index 6712302..cb1371f
--- a/src/backend/executor/execUtils.c
+++ b/src/backend/executor/execUtils.c
@@ -1200,6 +1200,34 @@ ExecGetReturningSlot(EState *estate, Res
}
/*
+ * Return a relInfo's all-NULL tuple slot for processing returning tuples.
+ *
+ * Note: this slot is intentionally filled with NULLs in every column, and
+ * should be considered read-only --- the caller must not update it.
+ */
+TupleTableSlot *
+ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo)
+{
+ if (relInfo->ri_AllNullSlot == NULL)
+ {
+ Relation rel = relInfo->ri_RelationDesc;
+ MemoryContext oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
+ TupleTableSlot *slot;
+
+ slot = ExecInitExtraTupleSlot(estate,
+ RelationGetDescr(rel),
+ table_slot_callbacks(rel));
+ ExecStoreAllNullTuple(slot);
+
+ relInfo->ri_AllNullSlot = slot;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ return relInfo->ri_AllNullSlot;
+}
+
+/*
* Return the map needed to convert given child result relation's tuples to
* the rowtype of the query's main target ("root") relation. Note that a
* NULL result is valid and means that no conversion is needed.
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
new file mode 100644
index 1161520..e84e47e
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -102,6 +102,13 @@ typedef struct ModifyTableContext
TM_FailureData tmfd;
/*
+ * The tuple deleted when doing a cross-partition UPDATE with a RETURNING
+ * clause that refers to OLD columns (converted to the root's tuple
+ * descriptor).
+ */
+ TupleTableSlot *cpDeletedSlot;
+
+ /*
* The tuple projected by the INSERT's RETURNING clause, when doing a
* cross-partition UPDATE
*/
@@ -243,34 +250,67 @@ ExecCheckPlanOutput(Relation resultRel,
/*
* ExecProcessReturning --- evaluate a RETURNING list
*
+ * context: context for the ModifyTable operation
* resultRelInfo: current result rel
- * tupleSlot: slot holding tuple actually inserted/updated/deleted
+ * cmdType: operation/merge action performed (INSERT, UPDATE, or DELETE)
+ * oldSlot: slot holding old tuple deleted or updated
+ * newSlot: slot holding new tuple inserted or updated
* planSlot: slot holding tuple returned by top subplan node
*
- * Note: If tupleSlot is NULL, the FDW should have already provided econtext's
- * scan tuple.
+ * Note: If oldSlot and newSlot are NULL, the FDW should have already provided
+ * econtext's scan tuple and its old & new tuples are not needed (FDW direct-
+ * modify is disabled if the RETURNING list refers to any OLD/NEW values).
*
* Returns a slot holding the result tuple
*/
static TupleTableSlot *
-ExecProcessReturning(ResultRelInfo *resultRelInfo,
- TupleTableSlot *tupleSlot,
+ExecProcessReturning(ModifyTableContext *context,
+ ResultRelInfo *resultRelInfo,
+ CmdType cmdType,
+ TupleTableSlot *oldSlot,
+ TupleTableSlot *newSlot,
TupleTableSlot *planSlot)
{
+ EState *estate = context->estate;
ProjectionInfo *projectReturning = resultRelInfo->ri_projectReturning;
ExprContext *econtext = projectReturning->pi_exprContext;
/* Make tuple and any needed join variables available to ExecProject */
- if (tupleSlot)
- econtext->ecxt_scantuple = tupleSlot;
+ if (cmdType == CMD_DELETE && oldSlot)
+ econtext->ecxt_scantuple = oldSlot;
+ if (cmdType != CMD_DELETE && newSlot)
+ econtext->ecxt_scantuple = newSlot;
econtext->ecxt_outertuple = planSlot;
+ /* Make old/new tuples available to ExecProject, if required */
+ if (oldSlot)
+ econtext->ecxt_oldtuple = oldSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ econtext->ecxt_oldtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_oldtuple = NULL; /* No references to OLD columns */
+
+ if (newSlot)
+ econtext->ecxt_newtuple = newSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW)
+ econtext->ecxt_newtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_newtuple = NULL; /* No references to NEW columns */
+
/*
- * RETURNING expressions might reference the tableoid column, so
- * reinitialize tts_tableOid before evaluating them.
+ * Tell ExecProject whether or not the OLD/NEW rows actually exist. This
+ * information is required to evaluate ReturningExpr nodes and also in
+ * ExecEvalSysVar and ExecEvalWholeRowVar.
*/
- econtext->ecxt_scantuple->tts_tableOid =
- RelationGetRelid(resultRelInfo->ri_RelationDesc);
+ if (oldSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_OLD_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_OLD_IS_NULL;
+
+ if (newSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_NEW_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_NEW_IS_NULL;
/* Compute the RETURNING expressions */
return ExecProject(projectReturning);
@@ -1204,7 +1244,56 @@ ExecInsert(ModifyTableContext *context,
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- result = ExecProcessReturning(resultRelInfo, slot, planSlot);
+ {
+ TupleTableSlot *oldSlot = NULL;
+
+ /*
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD columns, ExecDelete() will have saved the tuple
+ * deleted from the original partition, which we must use here to
+ * compute the OLD column values. Otherwise, all OLD column values
+ * will be NULL.
+ */
+ if (context->cpDeletedSlot)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the OLD tuple to the new partition's format/slot, if
+ * needed. Note that ExceDelete() already converted it to the
+ * root's partition's format/slot.
+ */
+ oldSlot = context->cpDeletedSlot;
+ tupconv_map = ExecGetRootToChildMap(resultRelInfo, estate);
+ if (tupconv_map != NULL)
+ {
+ oldSlot = execute_attr_map_slot(tupconv_map->attrMap,
+ oldSlot,
+ ExecGetReturningSlot(estate,
+ resultRelInfo));
+
+ oldSlot->tts_tableOid = context->cpDeletedSlot->tts_tableOid;
+ ItemPointerCopy(&context->cpDeletedSlot->tts_tid, &oldSlot->tts_tid);
+ }
+ }
+
+ result = ExecProcessReturning(context, resultRelInfo, CMD_INSERT,
+ oldSlot, slot, planSlot);
+
+ /*
+ * For a cross-partition UPDATE, release the old tuple, first making
+ * sure that the result slot has a local copy of any pass-by-reference
+ * values.
+ */
+ if (context->cpDeletedSlot)
+ {
+ ExecMaterializeSlot(result);
+ ExecClearTuple(oldSlot);
+ if (context->cpDeletedSlot != oldSlot)
+ ExecClearTuple(context->cpDeletedSlot);
+ context->cpDeletedSlot = NULL;
+ }
+ }
if (inserted_tuple)
*inserted_tuple = slot;
@@ -1442,6 +1531,7 @@ ExecDelete(ModifyTableContext *context,
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
TupleTableSlot *slot = NULL;
TM_Result result;
+ bool saveOld;
if (tupleDeleted)
*tupleDeleted = false;
@@ -1676,8 +1766,17 @@ ldelete:
ExecDeleteEpilogue(context, resultRelInfo, tupleid, oldtuple, changingPart);
- /* Process RETURNING if present and if requested */
- if (processReturning && resultRelInfo->ri_projectReturning)
+ /*
+ * Process RETURNING if present and if requested.
+ *
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD column values, save the old tuple here for later
+ * processing of the RETURNING list by ExecInsert().
+ */
+ saveOld = changingPart && resultRelInfo->ri_projectReturning &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD;
+
+ if (resultRelInfo->ri_projectReturning && (processReturning || saveOld))
{
/*
* We have to put the target tuple into a slot, which means first we
@@ -1705,7 +1804,41 @@ ldelete:
}
}
- rslot = ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ /*
+ * If required, save the old tuple for later processing of the
+ * RETURNING list by ExecInsert().
+ */
+ if (saveOld)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the tuple into the root partition's format/slot, if
+ * needed. ExecInsert() will then convert it to the new
+ * partition's format/slot, if necessary.
+ */
+ tupconv_map = ExecGetChildToRootMap(resultRelInfo);
+ if (tupconv_map != NULL)
+ {
+ ResultRelInfo *rootRelInfo = context->mtstate->rootResultRelInfo;
+ TupleTableSlot *oldSlot = slot;
+
+ slot = execute_attr_map_slot(tupconv_map->attrMap,
+ slot,
+ ExecGetReturningSlot(estate,
+ rootRelInfo));
+
+ slot->tts_tableOid = oldSlot->tts_tableOid;
+ ItemPointerCopy(&oldSlot->tts_tid, &slot->tts_tid);
+ }
+
+ context->cpDeletedSlot = slot;
+
+ return NULL;
+ }
+
+ rslot = ExecProcessReturning(context, resultRelInfo, CMD_DELETE,
+ slot, NULL, context->planSlot);
/*
* Before releasing the target tuple again, make sure rslot has a
@@ -1758,6 +1891,7 @@ ExecCrossPartitionUpdate(ModifyTableCont
bool tuple_deleted;
TupleTableSlot *epqslot = NULL;
+ context->cpDeletedSlot = NULL;
context->cpUpdateReturningSlot = NULL;
*retry_slot = NULL;
@@ -2258,6 +2392,7 @@ ExecCrossPartitionUpdateForeignKey(Modif
* the planSlot. oldtuple is passed to foreign table triggers; it is
* NULL when the foreign table has no relevant triggers.
*
+ * oldSlot contains the old tuple value.
* slot contains the new tuple value to be stored.
* planSlot is the output of the ModifyTable's subplan; we use it
* to access values from other input tables (for RETURNING),
@@ -2270,8 +2405,8 @@ ExecCrossPartitionUpdateForeignKey(Modif
*/
static TupleTableSlot *
ExecUpdate(ModifyTableContext *context, ResultRelInfo *resultRelInfo,
- ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot,
- bool canSetTag)
+ ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *oldSlot,
+ TupleTableSlot *slot, bool canSetTag)
{
EState *estate = context->estate;
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
@@ -2389,7 +2524,6 @@ redo_act:
{
TupleTableSlot *inputslot;
TupleTableSlot *epqslot;
- TupleTableSlot *oldSlot;
if (IsolationUsesXactSnapshot())
ereport(ERROR,
@@ -2504,7 +2638,8 @@ redo_act:
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- return ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ return ExecProcessReturning(context, resultRelInfo, CMD_UPDATE,
+ oldSlot, slot, context->planSlot);
return NULL;
}
@@ -2724,16 +2859,23 @@ ExecOnConflictUpdate(ModifyTableContext
/* Execute UPDATE with projection */
*returning = ExecUpdate(context, resultRelInfo,
- conflictTid, NULL,
+ conflictTid, NULL, existing,
resultRelInfo->ri_onConflict->oc_ProjSlot,
canSetTag);
/*
* Clear out existing tuple, as there might not be another conflict among
* the next input rows. Don't want to hold resources till the end of the
- * query.
+ * query. First though, make sure that the returning slot, if any, has a
+ * local copy of any OLD pass-by-reference values, if it refers to any OLD
+ * columns.
*/
+ if (*returning != NULL &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ ExecMaterializeSlot(*returning);
+
ExecClearTuple(existing);
+
return true;
}
@@ -3338,13 +3480,20 @@ lmerge_matched:
switch (commandType)
{
case CMD_UPDATE:
- rslot = ExecProcessReturning(resultRelInfo, newslot,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_UPDATE,
+ resultRelInfo->ri_oldTupleSlot,
+ newslot,
context->planSlot);
break;
case CMD_DELETE:
- rslot = ExecProcessReturning(resultRelInfo,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_DELETE,
resultRelInfo->ri_oldTupleSlot,
+ NULL,
context->planSlot);
break;
@@ -3894,6 +4043,7 @@ ExecModifyTable(PlanState *pstate)
if (node->mt_merge_pending_not_matched != NULL)
{
context.planSlot = node->mt_merge_pending_not_matched;
+ context.cpDeletedSlot = NULL;
slot = ExecMergeNotMatched(&context, node->resultRelInfo,
node->canSetTag);
@@ -3913,6 +4063,7 @@ ExecModifyTable(PlanState *pstate)
/* Fetch the next row from subplan */
context.planSlot = ExecProcNode(subplanstate);
+ context.cpDeletedSlot = NULL;
/* No more tuples to process? */
if (TupIsNull(context.planSlot))
@@ -3980,9 +4131,15 @@ ExecModifyTable(PlanState *pstate)
* A scan slot containing the data that was actually inserted,
* updated or deleted has already been made available to
* ExecProcessReturning by IterateDirectModify, so no need to
- * provide it here.
+ * provide it here. The individual old and new slots are not
+ * needed, since direct-modify is disabled if the RETURNING list
+ * refers to OLD/NEW values.
*/
- slot = ExecProcessReturning(resultRelInfo, NULL, context.planSlot);
+ Assert((resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD) == 0 &&
+ (resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW) == 0);
+
+ slot = ExecProcessReturning(&context, resultRelInfo, operation,
+ NULL, NULL, context.planSlot);
return slot;
}
@@ -4172,7 +4329,7 @@ ExecModifyTable(PlanState *pstate)
/* Now apply the update. */
slot = ExecUpdate(&context, resultRelInfo, tupleid, oldtuple,
- slot, node->canSetTag);
+ oldSlot, slot, node->canSetTag);
if (tuplock)
UnlockTuple(resultRelInfo->ri_RelationDesc, tupleid,
InplaceUpdateTupleLock);
diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c
new file mode 100644
index 48ccdb9..909c924
--- a/src/backend/jit/llvm/llvmjit_expr.c
+++ b/src/backend/jit/llvm/llvmjit_expr.c
@@ -105,6 +105,8 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_innerslot;
LLVMValueRef v_outerslot;
LLVMValueRef v_scanslot;
+ LLVMValueRef v_oldslot;
+ LLVMValueRef v_newslot;
LLVMValueRef v_resultslot;
/* nulls/values of slots */
@@ -114,6 +116,10 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_outernulls;
LLVMValueRef v_scanvalues;
LLVMValueRef v_scannulls;
+ LLVMValueRef v_oldvalues;
+ LLVMValueRef v_oldnulls;
+ LLVMValueRef v_newvalues;
+ LLVMValueRef v_newnulls;
LLVMValueRef v_resultvalues;
LLVMValueRef v_resultnulls;
@@ -200,6 +206,16 @@ llvm_compile_expr(ExprState *state)
v_econtext,
FIELDNO_EXPRCONTEXT_OUTERTUPLE,
"v_outerslot");
+ v_oldslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_OLDTUPLE,
+ "v_oldslot");
+ v_newslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_NEWTUPLE,
+ "v_newslot");
v_resultslot = l_load_struct_gep(b,
StructExprState,
v_state,
@@ -237,6 +253,26 @@ llvm_compile_expr(ExprState *state)
v_outerslot,
FIELDNO_TUPLETABLESLOT_ISNULL,
"v_outernulls");
+ v_oldvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_oldvalues");
+ v_oldnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_oldnulls");
+ v_newvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_newvalues");
+ v_newnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_newnulls");
v_resultvalues = l_load_struct_gep(b,
StructTupleTableSlot,
v_resultslot,
@@ -302,6 +338,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_FETCHSOME:
case EEOP_OUTER_FETCHSOME:
case EEOP_SCAN_FETCHSOME:
+ case EEOP_OLD_FETCHSOME:
+ case EEOP_NEW_FETCHSOME:
{
TupleDesc desc = NULL;
LLVMValueRef v_slot;
@@ -326,8 +364,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_FETCHSOME)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_FETCHSOME)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_FETCHSOME)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
/*
* Check if all required attributes are available, or
@@ -396,6 +438,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_VAR:
case EEOP_OUTER_VAR:
case EEOP_SCAN_VAR:
+ case EEOP_OLD_VAR:
+ case EEOP_NEW_VAR:
{
LLVMValueRef value,
isnull;
@@ -413,11 +457,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
v_attnum = l_int32_const(lc, op->d.var.attnum);
value = l_load_gep1(b, TypeSizeT, v_values, v_attnum, "");
@@ -432,6 +486,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_SYSVAR:
case EEOP_OUTER_SYSVAR:
case EEOP_SCAN_SYSVAR:
+ case EEOP_OLD_SYSVAR:
+ case EEOP_NEW_SYSVAR:
{
LLVMValueRef v_slot;
@@ -439,8 +495,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_SYSVAR)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_SYSVAR)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_SYSVAR)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
build_EvalXFunc(b, mod, "ExecEvalSysVar",
v_state, op, v_econtext, v_slot);
@@ -458,6 +518,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_ASSIGN_INNER_VAR:
case EEOP_ASSIGN_OUTER_VAR:
case EEOP_ASSIGN_SCAN_VAR:
+ case EEOP_ASSIGN_OLD_VAR:
+ case EEOP_ASSIGN_NEW_VAR:
{
LLVMValueRef v_value;
LLVMValueRef v_isnull;
@@ -478,11 +540,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_ASSIGN_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_ASSIGN_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
/* load data */
v_attnum = l_int32_const(lc, op->d.assign_var.attnum);
@@ -1639,6 +1711,45 @@ llvm_compile_expr(ExprState *state)
LLVMBuildBr(b, opblocks[opno + 1]);
break;
+ case EEOP_RETURNINGEXPR:
+ {
+ LLVMBasicBlockRef b_isnull;
+ LLVMValueRef v_flagsp;
+ LLVMValueRef v_flags;
+ LLVMValueRef v_nullflag;
+
+ b_isnull = l_bb_before_v(opblocks[opno + 1],
+ "op.%d.row.isnull", opno);
+
+ /*
+ * The next op actually evaluates the expression. If the
+ * OLD/NEW row doesn't exist, skip that and return NULL.
+ */
+ v_flagsp = l_struct_gep(b,
+ StructExprState,
+ v_state,
+ FIELDNO_EXPRSTATE_FLAGS,
+ "v.state.flags");
+ v_flags = l_load(b, TypeStorageBool, v_flagsp, "");
+
+ v_nullflag = l_int8_const(lc, op->d.returningexpr.nullflag);
+
+ LLVMBuildCondBr(b,
+ LLVMBuildICmp(b, LLVMIntEQ,
+ LLVMBuildAnd(b, v_flags,
+ v_nullflag, ""),
+ l_sbool_const(0), ""),
+ opblocks[opno + 1], b_isnull);
+
+ LLVMPositionBuilderAtEnd(b, b_isnull);
+
+ LLVMBuildStore(b, l_sizet_const(0), v_resvaluep);
+ LLVMBuildStore(b, l_sbool_const(1), v_resnullp);
+
+ LLVMBuildBr(b, opblocks[op->d.returningexpr.jumpdone]);
+ break;
+ }
+
case EEOP_ARRAYEXPR:
build_EvalXFunc(b, mod, "ExecEvalArrayExpr",
v_state, op);
diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c
new file mode 100644
index 9cac3c1..4e25ca6
--- a/src/backend/nodes/makefuncs.c
+++ b/src/backend/nodes/makefuncs.c
@@ -80,12 +80,14 @@ makeVar(int varno,
var->varlevelsup = varlevelsup;
/*
- * Only a few callers need to make Var nodes with non-null varnullingrels,
- * or with varnosyn/varattnosyn different from varno/varattno. We don't
- * provide separate arguments for them, but just initialize them to NULL
- * and the given varno/varattno. This reduces code clutter and chance of
- * error for most callers.
+ * Only a few callers need to make Var nodes with varreturningtype
+ * different from VAR_RETURNING_DEFAULT, non-null varnullingrels, or with
+ * varnosyn/varattnosyn different from varno/varattno. We don't provide
+ * separate arguments for them, but just initialize them to sensible
+ * default values. This reduces code clutter and chance of error for most
+ * callers.
*/
+ var->varreturningtype = VAR_RETURNING_DEFAULT;
var->varnullingrels = NULL;
var->varnosyn = (Index) varno;
var->varattnosyn = varattno;
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
new file mode 100644
index 0d00e02..04df2db
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -278,6 +278,9 @@ exprType(const Node *expr)
type = exprType((Node *) n->expr);
}
break;
+ case T_ReturningExpr:
+ type = exprType((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
type = exprType((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -529,6 +532,8 @@ exprTypmod(const Node *expr)
return ((const CoerceToDomainValue *) expr)->typeMod;
case T_SetToDefault:
return ((const SetToDefault *) expr)->typeMod;
+ case T_ReturningExpr:
+ return exprTypmod((Node *) ((const ReturningExpr *) expr)->retexpr);
case T_PlaceHolderVar:
return exprTypmod((Node *) ((const PlaceHolderVar *) expr)->phexpr);
default:
@@ -1047,6 +1052,9 @@ exprCollation(const Node *expr)
case T_InferenceElem:
coll = exprCollation((Node *) ((const InferenceElem *) expr)->expr);
break;
+ case T_ReturningExpr:
+ coll = exprCollation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
coll = exprCollation((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -1298,6 +1306,10 @@ exprSetCollation(Node *expr, Oid collati
/* NextValueExpr's result is an integer type ... */
Assert(!OidIsValid(collation)); /* ... so never set a collation */
break;
+ case T_ReturningExpr:
+ exprSetCollation((Node *) ((ReturningExpr *) expr)->retexpr,
+ collation);
+ break;
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(expr));
break;
@@ -1624,6 +1636,9 @@ exprLocation(const Node *expr)
case T_SetToDefault:
loc = ((const SetToDefault *) expr)->location;
break;
+ case T_ReturningExpr:
+ loc = exprLocation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_TargetEntry:
/* just use argument's location */
loc = exprLocation((Node *) ((const TargetEntry *) expr)->expr);
@@ -2614,6 +2629,8 @@ expression_tree_walker_impl(Node *node,
return WALK(((PlaceHolderVar *) node)->phexpr);
case T_InferenceElem:
return WALK(((InferenceElem *) node)->expr);
+ case T_ReturningExpr:
+ return WALK(((ReturningExpr *) node)->retexpr);
case T_AppendRelInfo:
{
AppendRelInfo *appinfo = (AppendRelInfo *) node;
@@ -3455,6 +3472,16 @@ expression_tree_mutator_impl(Node *node,
return (Node *) newnode;
}
break;
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ ReturningExpr *newnode;
+
+ FLATCOPY(newnode, rexpr, ReturningExpr);
+ MUTATE(newnode->retexpr, rexpr->retexpr, Expr *);
+ return (Node *) newnode;
+ }
+ break;
case T_TargetEntry:
{
TargetEntry *targetentry = (TargetEntry *) node;
@@ -4006,6 +4033,7 @@ raw_expression_tree_walker_impl(Node *no
case T_A_Const:
case T_A_Star:
case T_MergeSupportFunc:
+ case T_ReturningOption:
/* primitive node types with no subnodes */
break;
case T_Alias:
@@ -4234,7 +4262,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->onConflictClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4250,7 +4278,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->whereClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4268,7 +4296,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->fromClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4286,7 +4314,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->mergeWhenClauses))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4304,6 +4332,16 @@ raw_expression_tree_walker_impl(Node *no
return true;
}
break;
+ case T_ReturningClause:
+ {
+ ReturningClause *returning = (ReturningClause *) node;
+
+ if (WALK(returning->options))
+ return true;
+ if (WALK(returning->exprs))
+ return true;
+ }
+ break;
case T_SelectStmt:
{
SelectStmt *stmt = (SelectStmt *) node;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
new file mode 100644
index 172edb6..6346c4e
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -3985,6 +3985,7 @@ subquery_push_qual(Query *subquery, Rang
*/
qual = ReplaceVarsFromTargetList(qual, rti, 0, rte,
subquery->targetList,
+ subquery->resultRelation,
REPLACEVARS_REPORT_ERROR, 0,
&subquery->hasSubLinks);
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
new file mode 100644
index c13586c..8cdf311
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -7113,6 +7113,8 @@ make_modifytable(PlannerInfo *root, Plan
int epqParam)
{
ModifyTable *node = makeNode(ModifyTable);
+ bool returning_old_or_new = false;
+ bool returning_old_or_new_valid = false;
List *fdw_private_list;
Bitmapset *direct_modify_plans;
ListCell *lc;
@@ -7177,6 +7179,8 @@ make_modifytable(PlannerInfo *root, Plan
}
node->updateColnosLists = updateColnosLists;
node->withCheckOptionLists = withCheckOptionLists;
+ node->returningOld = root->parse->returningOld;
+ node->returningNew = root->parse->returningNew;
node->returningLists = returningLists;
node->rowMarks = rowMarks;
node->mergeActionLists = mergeActionLists;
@@ -7257,7 +7261,8 @@ make_modifytable(PlannerInfo *root, Plan
* callback functions needed for that and (2) there are no local
* structures that need to be run for each modified row: row-level
* triggers on the foreign table, stored generated columns, WITH CHECK
- * OPTIONs from parent views.
+ * OPTIONs from parent views, or Vars returning OLD/NEW in the
+ * RETURNING list.
*/
direct_modify = false;
if (fdwroutine != NULL &&
@@ -7268,7 +7273,18 @@ make_modifytable(PlannerInfo *root, Plan
withCheckOptionLists == NIL &&
!has_row_triggers(root, rti, operation) &&
!has_stored_generated_columns(root, rti))
- direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ {
+ /* returning_old_or_new is the same for all result relations */
+ if (!returning_old_or_new_valid)
+ {
+ returning_old_or_new =
+ contain_vars_returning_old_or_new((Node *)
+ root->parse->returningList);
+ returning_old_or_new_valid = true;
+ }
+ if (!returning_old_or_new)
+ direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ }
if (direct_modify)
direct_modify_plans = bms_add_member(direct_modify_plans, i);
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
new file mode 100644
index 91c7c4f..218e46a
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -3074,6 +3074,21 @@ fix_join_expr_mutator(Node *node, fix_jo
{
Var *var = (Var *) node;
+ /*
+ * Verify that Vars with non-default varreturningtype only appear in
+ * the RETURNING list, and refer to the target relation.
+ */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ if (context->inner_itlist != NULL ||
+ context->outer_itlist == NULL ||
+ context->acceptable_rel == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+ if (var->varno != context->acceptable_rel)
+ elog(ERROR, "wrong varno %d (expected %d) for variable returning old/new",
+ var->varno, context->acceptable_rel);
+ }
+
/* Look for the var in the input tlists, first in the outer */
if (context->outer_itlist)
{
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
new file mode 100644
index 6d003cc..0118876
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -354,17 +354,19 @@ build_subplan(PlannerInfo *root, Plan *p
Node *arg = pitem->item;
/*
- * The Var, PlaceHolderVar, Aggref or GroupingFunc has already been
- * adjusted to have the correct varlevelsup, phlevelsup, or
- * agglevelsup.
+ * The Var, PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr has
+ * already been adjusted to have the correct varlevelsup, phlevelsup,
+ * agglevelsup, or retlevelsup.
*
- * If it's a PlaceHolderVar, Aggref or GroupingFunc, its arguments
- * might contain SubLinks, which have not yet been processed (see the
- * comments for SS_replace_correlation_vars). Do that now.
+ * If it's a PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr,
+ * its arguments might contain SubLinks, which have not yet been
+ * processed (see the comments for SS_replace_correlation_vars). Do
+ * that now.
*/
if (IsA(arg, PlaceHolderVar) ||
IsA(arg, Aggref) ||
- IsA(arg, GroupingFunc))
+ IsA(arg, GroupingFunc) ||
+ IsA(arg, ReturningExpr))
arg = SS_process_sublinks(root, arg, false);
splan->parParam = lappend_int(splan->parParam, pitem->paramId);
@@ -1842,8 +1844,8 @@ convert_EXISTS_to_ANY(PlannerInfo *root,
/*
* Replace correlation vars (uplevel vars) with Params.
*
- * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions, and
- * MergeSupportFuncs are replaced, too.
+ * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions,
+ * MergeSupportFuncs, and ReturningExprs are replaced, too.
*
* Note: it is critical that this runs immediately after SS_process_sublinks.
* Since we do not recurse into the arguments of uplevel PHVs and aggregates,
@@ -1903,6 +1905,12 @@ replace_correlation_vars_mutator(Node *n
return (Node *) replace_outer_merge_support(root,
(MergeSupportFunc *) node);
}
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return (Node *) replace_outer_returning(root,
+ (ReturningExpr *) node);
+ }
return expression_tree_mutator(node,
replace_correlation_vars_mutator,
(void *) root);
@@ -1958,11 +1966,11 @@ process_sublinks_mutator(Node *node, pro
}
/*
- * Don't recurse into the arguments of an outer PHV, Aggref or
- * GroupingFunc here. Any SubLinks in the arguments have to be dealt with
- * at the outer query level; they'll be handled when build_subplan
- * collects the PHV, Aggref or GroupingFunc into the arguments to be
- * passed down to the current subplan.
+ * Don't recurse into the arguments of an outer PHV, Aggref, GroupingFunc
+ * or ReturningExpr here. Any SubLinks in the arguments have to be dealt
+ * with at the outer query level; they'll be handled when build_subplan
+ * collects the PHV, Aggref, GroupingFunc or ReturningExpr into the
+ * arguments to be passed down to the current subplan.
*/
if (IsA(node, PlaceHolderVar))
{
@@ -1979,6 +1987,11 @@ process_sublinks_mutator(Node *node, pro
if (((GroupingFunc *) node)->agglevelsup > 0)
return node;
}
+ else if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return node;
+ }
/*
* We should never see a SubPlan expression in the input (since this is
@@ -2091,7 +2104,9 @@ SS_identify_outer_params(PlannerInfo *ro
outer_params = NULL;
for (proot = root->parent_root; proot != NULL; proot = proot->parent_root)
{
- /* Include ordinary Var/PHV/Aggref/GroupingFunc params */
+ /*
+ * Include ordinary Var/PHV/Aggref/GroupingFunc/ReturningExpr params.
+ */
foreach(l, proot->plan_params)
{
PlannerParamItem *pitem = (PlannerParamItem *) lfirst(l);
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
new file mode 100644
index 4d7f972..79d3e99
--- a/src/backend/optimizer/prep/prepjointree.c
+++ b/src/backend/optimizer/prep/prepjointree.c
@@ -2512,7 +2512,8 @@ pullup_replace_vars_callback(Var *var,
* expansion with varlevelsup = 0, and then adjust below if needed.
*/
expandRTE(rcon->target_rte,
- var->varno, 0 /* not varlevelsup */ , var->location,
+ var->varno, 0 /* not varlevelsup */ ,
+ var->varreturningtype, var->location,
(var->vartype != RECORDOID),
&colnames, &fields);
/* Expand the generated per-field Vars, but don't insert PHVs there */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
new file mode 100644
index 4989722..7a6fe58
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -253,6 +253,13 @@ adjust_appendrel_attrs_mutator(Node *nod
* all non-Var outputs of such subqueries, and then we could look up
* the pre-existing PHV here. Or perhaps just wrap the translations
* that way to begin with?
+ *
+ * If var->varreturningtype is not VAR_RETURNING_DEFAULT, then that
+ * also needs to be copied to the translated Var. That too would fail
+ * if the translation wasn't a Var, but that should never happen since
+ * a non-default var->varreturningtype is only used for Vars referring
+ * to the result relation, which should never be a flattened UNION ALL
+ * subquery.
*/
for (cnt = 0; cnt < nappinfos; cnt++)
@@ -283,9 +290,17 @@ adjust_appendrel_attrs_mutator(Node *nod
elog(ERROR, "attribute %d of relation \"%s\" does not exist",
var->varattno, get_rel_name(appinfo->parent_reloid));
if (IsA(newnode, Var))
+ {
+ ((Var *) newnode)->varreturningtype = var->varreturningtype;
((Var *) newnode)->varnullingrels = var->varnullingrels;
- else if (var->varnullingrels != NULL)
- elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
+ else
+ {
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
+ if (var->varnullingrels != NULL)
+ elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
return newnode;
}
else if (var->varattno == 0)
@@ -339,6 +354,8 @@ adjust_appendrel_attrs_mutator(Node *nod
rowexpr->colnames = copyObject(rte->eref->colnames);
rowexpr->location = -1;
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
if (var->varnullingrels != NULL)
elog(ERROR, "failed to apply nullingrels to a non-Var");
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
new file mode 100644
index b4e085e..09a1ea1
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -1296,6 +1296,7 @@ contain_leaked_vars_walker(Node *node, v
case T_NullTest:
case T_BooleanTest:
case T_NextValueExpr:
+ case T_ReturningExpr:
case T_List:
/*
@@ -3393,6 +3394,8 @@ eval_const_expressions_mutator(Node *nod
fselect->resulttypmod,
fselect->resultcollid,
((Var *) arg)->varlevelsup);
+ /* New Var has same OLD/NEW returning as old one */
+ newvar->varreturningtype = ((Var *) arg)->varreturningtype;
/* New Var is nullable by same rels as the old one */
newvar->varnullingrels = ((Var *) arg)->varnullingrels;
return (Node *) newvar;
diff --git a/src/backend/optimizer/util/paramassign.c b/src/backend/optimizer/util/paramassign.c
new file mode 100644
index f461fed..38a3986
--- a/src/backend/optimizer/util/paramassign.c
+++ b/src/backend/optimizer/util/paramassign.c
@@ -91,6 +91,7 @@ assign_param_for_var(PlannerInfo *root,
pvar->vartype == var->vartype &&
pvar->vartypmod == var->vartypmod &&
pvar->varcollid == var->varcollid &&
+ pvar->varreturningtype == var->varreturningtype &&
bms_equal(pvar->varnullingrels, var->varnullingrels))
return pitem->paramId;
}
@@ -357,6 +358,52 @@ replace_outer_merge_support(PlannerInfo
return retval;
}
+
+/*
+ * Generate a Param node to replace the given ReturningExpr expression which
+ * is expected to have retlevelsup > 0 (ie, it is not local). Record the need
+ * for the ReturningExpr in the proper upper-level root->plan_params.
+ */
+Param *
+replace_outer_returning(PlannerInfo *root, ReturningExpr *rexpr)
+{
+ Param *retval;
+ PlannerParamItem *pitem;
+ Index levelsup;
+ Oid ptype = exprType((Node *) rexpr->retexpr);
+
+ Assert(rexpr->retlevelsup > 0 && rexpr->retlevelsup < root->query_level);
+
+ /* Find the query level the ReturningExpr belongs to */
+ for (levelsup = rexpr->retlevelsup; levelsup > 0; levelsup--)
+ root = root->parent_root;
+
+ /*
+ * It does not seem worthwhile to try to de-duplicate references to outer
+ * ReturningExprs. Just make a new slot every time.
+ */
+ rexpr = copyObject(rexpr);
+ IncrementVarSublevelsUp((Node *) rexpr, -((int) rexpr->retlevelsup), 0);
+ Assert(rexpr->retlevelsup == 0);
+
+ pitem = makeNode(PlannerParamItem);
+ pitem->item = (Node *) rexpr;
+ pitem->paramId = list_length(root->glob->paramExecTypes);
+ root->glob->paramExecTypes = lappend_oid(root->glob->paramExecTypes,
+ ptype);
+
+ root->plan_params = lappend(root->plan_params, pitem);
+
+ retval = makeNode(Param);
+ retval->paramkind = PARAM_EXEC;
+ retval->paramid = pitem->paramId;
+ retval->paramtype = ptype;
+ retval->paramtypmod = exprTypmod((Node *) rexpr->retexpr);
+ retval->paramcollid = exprCollation((Node *) rexpr->retexpr);
+ retval->location = exprLocation((Node *) rexpr->retexpr);
+
+ return retval;
+}
/*
* Generate a Param node to replace the given Var,
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
new file mode 100644
index b913f91..16e6353
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1843,8 +1843,8 @@ build_physical_tlist(PlannerInfo *root,
case RTE_NAMEDTUPLESTORE:
case RTE_RESULT:
/* Not all of these can have dropped cols, but share code anyway */
- expandRTE(rte, varno, 0, -1, true /* include dropped */ ,
- NULL, &colvars);
+ expandRTE(rte, varno, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , NULL, &colvars);
foreach(l, colvars)
{
var = (Var *) lfirst(l);
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
new file mode 100644
index f7534ad..4b50767
--- a/src/backend/optimizer/util/var.c
+++ b/src/backend/optimizer/util/var.c
@@ -76,6 +76,7 @@ static bool pull_varattnos_walker(Node *
static bool pull_vars_walker(Node *node, pull_vars_context *context);
static bool contain_var_clause_walker(Node *node, void *context);
static bool contain_vars_of_level_walker(Node *node, int *sublevels_up);
+static bool contain_vars_returning_old_or_new_walker(Node *node, void *context);
static bool locate_var_of_level_walker(Node *node,
locate_var_of_level_context *context);
static bool pull_var_clause_walker(Node *node,
@@ -495,6 +496,49 @@ contain_vars_of_level_walker(Node *node,
}
+/*
+ * contain_vars_returning_old_or_new
+ * Recursively scan a clause to discover whether it contains any Var nodes
+ * (of the current query level) whose varreturningtype is VAR_RETURNING_OLD
+ * or VAR_RETURNING_NEW.
+ *
+ * Returns true if any found.
+ *
+ * Any ReturningExprs are also detected --- if an OLD/NEW Var was rewritten,
+ * we still regard this as a clause that returns OLD/NEW values.
+ *
+ * Does not examine subqueries, therefore must only be used after reduction
+ * of sublinks to subplans!
+ */
+bool
+contain_vars_returning_old_or_new(Node *node)
+{
+ return contain_vars_returning_old_or_new_walker(node, NULL);
+}
+
+static bool
+contain_vars_returning_old_or_new_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ if (((Var *) node)->varlevelsup == 0 &&
+ ((Var *) node)->varreturningtype != VAR_RETURNING_DEFAULT)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup == 0)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ return expression_tree_walker(node, contain_vars_returning_old_or_new_walker,
+ context);
+}
+
+
/*
* locate_var_of_level
* Find the parse location of any Var of the specified query level.
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
new file mode 100644
index e901203..73f1015
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -556,8 +556,8 @@ transformDeleteStmt(ParseState *pstate,
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -969,7 +969,7 @@ transformInsertStmt(ParseState *pstate,
* contain only the target relation, removing any entries added in a
* sub-SELECT or VALUES list.
*/
- if (stmt->onConflictClause || stmt->returningList)
+ if (stmt->onConflictClause || stmt->returningClause)
{
pstate->p_namespace = NIL;
addNSItemToQuery(pstate, pstate->p_target_nsitem,
@@ -982,10 +982,9 @@ transformInsertStmt(ParseState *pstate,
stmt->onConflictClause);
/* Process RETURNING, if any. */
- if (stmt->returningList)
- qry->returningList = transformReturningList(pstate,
- stmt->returningList,
- EXPR_KIND_RETURNING);
+ if (stmt->returningClause)
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -2462,8 +2461,8 @@ transformUpdateStmt(ParseState *pstate,
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/*
* Now we are done with SELECT-like processing, and can get on with
@@ -2559,18 +2558,120 @@ transformUpdateTargetList(ParseState *ps
}
/*
- * transformReturningList -
+ * addNSItemForReturning -
+ * add a ParseNamespaceItem for the OLD or NEW alias in RETURNING.
+ */
+static void
+addNSItemForReturning(ParseState *pstate, const char *aliasname,
+ VarReturningType returning_type)
+{
+ List *colnames;
+ int numattrs;
+ ParseNamespaceColumn *nscolumns;
+ ParseNamespaceItem *nsitem;
+
+ /* copy per-column data from the target relation */
+ colnames = pstate->p_target_nsitem->p_rte->eref->colnames;
+ numattrs = list_length(colnames);
+
+ nscolumns = (ParseNamespaceColumn *)
+ palloc(numattrs * sizeof(ParseNamespaceColumn));
+
+ memcpy(nscolumns, pstate->p_target_nsitem->p_nscolumns,
+ numattrs * sizeof(ParseNamespaceColumn));
+
+ /* mark all columns as returning OLD/NEW */
+ for (int i = 0; i < numattrs; i++)
+ nscolumns[i].p_varreturningtype = returning_type;
+
+ /* build the nsitem, copying most fields from the target relation */
+ nsitem = (ParseNamespaceItem *) palloc(sizeof(ParseNamespaceItem));
+ nsitem->p_names = makeAlias(aliasname, colnames);
+ nsitem->p_rte = pstate->p_target_nsitem->p_rte;
+ nsitem->p_rtindex = pstate->p_target_nsitem->p_rtindex;
+ nsitem->p_perminfo = pstate->p_target_nsitem->p_perminfo;
+ nsitem->p_nscolumns = nscolumns;
+ nsitem->p_returning_type = returning_type;
+
+ /* add it to the query namespace as a table-only item */
+ addNSItemToQuery(pstate, nsitem, false, true, false);
+}
+
+/*
+ * transformReturningClause -
* handle a RETURNING clause in INSERT/UPDATE/DELETE/MERGE
*/
-List *
-transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind)
+void
+transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind)
{
- List *rlist;
+ int save_nslen;
int save_next_resno;
- if (returningList == NIL)
- return NIL; /* nothing to do */
+ if (returningClause == NULL)
+ return; /* nothing to do */
+
+ /*
+ * Scan RETURNING WITH(...) options for OLD/NEW alias names. Complain if
+ * there is any conflict with existing relations.
+ */
+ foreach_node(ReturningOption, option, returningClause->options)
+ {
+ switch (option->option)
+ {
+ case RETURNING_OPTION_OLD:
+ if (qry->returningOld != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "OLD"),
+ parser_errposition(pstate, option->location));
+ qry->returningOld = option->value;
+ break;
+
+ case RETURNING_OPTION_NEW:
+ if (qry->returningNew != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "NEW"),
+ parser_errposition(pstate, option->location));
+ qry->returningNew = option->value;
+ break;
+
+ default:
+ elog(ERROR, "unrecognized returning option: %d", option->option);
+ }
+
+ if (refnameNamespaceItem(pstate, NULL, option->value, -1, NULL) != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_ALIAS),
+ errmsg("table name \"%s\" specified more than once",
+ option->value),
+ parser_errposition(pstate, option->location));
+ }
+
+ /*
+ * If no OLD/NEW aliases specified, use "old"/"new" unless masked by
+ * existing relations.
+ */
+ if (qry->returningOld == NULL &&
+ refnameNamespaceItem(pstate, NULL, "old", -1, NULL) == NULL)
+ qry->returningOld = "old";
+ if (qry->returningNew == NULL &&
+ refnameNamespaceItem(pstate, NULL, "new", -1, NULL) == NULL)
+ qry->returningNew = "new";
+
+ /*
+ * Add the OLD and NEW aliases to the query namespace, for use in
+ * expressions in the RETURNING list.
+ */
+ save_nslen = list_length(pstate->p_namespace);
+ if (qry->returningOld != NULL)
+ addNSItemForReturning(pstate, qry->returningOld, VAR_RETURNING_OLD);
+ if (qry->returningNew != NULL)
+ addNSItemForReturning(pstate, qry->returningNew, VAR_RETURNING_NEW);
/*
* We need to assign resnos starting at one in the RETURNING list. Save
@@ -2580,8 +2681,10 @@ transformReturningList(ParseState *pstat
save_next_resno = pstate->p_next_resno;
pstate->p_next_resno = 1;
- /* transform RETURNING identically to a SELECT targetlist */
- rlist = transformTargetList(pstate, returningList, exprKind);
+ /* transform RETURNING expressions identically to a SELECT targetlist */
+ qry->returningList = transformTargetList(pstate,
+ returningClause->exprs,
+ exprKind);
/*
* Complain if the nonempty tlist expanded to nothing (which is possible
@@ -2589,24 +2692,23 @@ transformReturningList(ParseState *pstat
* allow this, the parsed Query will look like it didn't have RETURNING,
* with results that would probably surprise the user.
*/
- if (rlist == NIL)
+ if (qry->returningList == NIL)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("RETURNING must have at least one column"),
parser_errposition(pstate,
- exprLocation(linitial(returningList)))));
+ exprLocation(linitial(returningClause->exprs)))));
/* mark column origins */
- markTargetListOrigins(pstate, rlist);
+ markTargetListOrigins(pstate, qry->returningList);
/* resolve any still-unresolved output columns as being type text */
if (pstate->p_resolve_unknowns)
- resolveTargetListUnknowns(pstate, rlist);
+ resolveTargetListUnknowns(pstate, qry->returningList);
/* restore state */
+ pstate->p_namespace = list_truncate(pstate->p_namespace, save_nslen);
pstate->p_next_resno = save_next_resno;
-
- return rlist;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
new file mode 100644
index 4aa8646..04672b4
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -278,6 +278,8 @@ static Node *makeRecursiveViewSelect(cha
MergeWhenClause *mergewhen;
struct KeyActions *keyactions;
struct KeyAction *keyaction;
+ ReturningClause *retclause;
+ ReturningOptionKind retoptionkind;
}
%type <node> stmt toplevel_stmt schema_stmt routine_body_stmt
@@ -447,7 +449,8 @@ static Node *makeRecursiveViewSelect(cha
opclass_purpose opt_opfamily transaction_mode_list_or_empty
OptTableFuncElementList TableFuncElementList opt_type_modifiers
prep_type_clause
- execute_param_clause using_clause returning_clause
+ execute_param_clause using_clause
+ returning_with_clause returning_options
opt_enum_val_list enum_val_list table_func_column_list
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
@@ -456,6 +459,9 @@ static Node *makeRecursiveViewSelect(cha
vacuum_relation_list opt_vacuum_relation_list
drop_option_list pub_obj_list
+%type <retclause> returning_clause
+%type <node> returning_option
+%type <retoptionkind> returning_option_kind
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
@@ -12168,7 +12174,7 @@ InsertStmt:
{
$5->relation = $4;
$5->onConflictClause = $6;
- $5->returningList = $7;
+ $5->returningClause = $7;
$5->withClause = $1;
$$ = (Node *) $5;
}
@@ -12301,8 +12307,45 @@ opt_conf_expr:
;
returning_clause:
- RETURNING target_list { $$ = $2; }
- | /* EMPTY */ { $$ = NIL; }
+ RETURNING returning_with_clause target_list
+ {
+ ReturningClause *n = makeNode(ReturningClause);
+
+ n->options = $2;
+ n->exprs = $3;
+ $$ = n;
+ }
+ | /* EMPTY */
+ {
+ $$ = NULL;
+ }
+ ;
+
+returning_with_clause:
+ WITH '(' returning_options ')' { $$ = $3; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+returning_options:
+ returning_option { $$ = list_make1($1); }
+ | returning_options ',' returning_option { $$ = lappend($1, $3); }
+ ;
+
+returning_option:
+ returning_option_kind AS ColId
+ {
+ ReturningOption *n = makeNode(ReturningOption);
+
+ n->option = $1;
+ n->value = $3;
+ n->location = @1;
+ $$ = (Node *) n;
+ }
+ ;
+
+returning_option_kind:
+ OLD { $$ = RETURNING_OPTION_OLD; }
+ | NEW { $$ = RETURNING_OPTION_NEW; }
;
@@ -12321,7 +12364,7 @@ DeleteStmt: opt_with_clause DELETE_P FRO
n->relation = $4;
n->usingClause = $5;
n->whereClause = $6;
- n->returningList = $7;
+ n->returningClause = $7;
n->withClause = $1;
$$ = (Node *) n;
}
@@ -12395,7 +12438,7 @@ UpdateStmt: opt_with_clause UPDATE relat
n->targetList = $5;
n->fromClause = $6;
n->whereClause = $7;
- n->returningList = $8;
+ n->returningClause = $8;
n->withClause = $1;
$$ = (Node *) n;
}
@@ -12473,7 +12516,7 @@ MergeStmt:
m->sourceRelation = $6;
m->joinCondition = $8;
m->mergeWhenClauses = $9;
- m->returningList = $10;
+ m->returningClause = $10;
$$ = (Node *) m;
}
diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c
new file mode 100644
index 8118036..a2b0753
--- a/src/backend/parser/parse_clause.c
+++ b/src/backend/parser/parse_clause.c
@@ -1587,6 +1587,7 @@ transformFromClauseItem(ParseState *psta
jnsitem->p_cols_visible = true;
jnsitem->p_lateral_only = false;
jnsitem->p_lateral_ok = true;
+ jnsitem->p_returning_type = VAR_RETURNING_DEFAULT;
/* Per SQL, we must check for alias conflicts */
checkNameSpaceConflicts(pstate, list_make1(jnsitem), my_namespace);
my_namespace = lappend(my_namespace, jnsitem);
@@ -1649,6 +1650,7 @@ buildVarFromNSColumn(ParseState *pstate,
nscol->p_varcollid,
0);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
new file mode 100644
index 36c1b7a..c8bbd38
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -2621,6 +2621,13 @@ transformWholeRowRef(ParseState *pstate,
* point, there seems no harm in expanding it now rather than during
* planning.
*
+ * Note that if the nsitem is an OLD/NEW alias for the target RTE (as can
+ * appear in a RETURNING list), its alias won't match the target RTE's
+ * alias, but we still want to make a whole-row Var here rather than a
+ * RowExpr, for consistency with direct references to the target RTE, and
+ * so that any dropped columns are handled correctly. Thus we also check
+ * p_returning_type here.
+ *
* Note that if the RTE is a function returning scalar, we create just a
* plain reference to the function value, not a composite containing a
* single column. This is pretty inconsistent at first sight, but it's
@@ -2628,13 +2635,17 @@ transformWholeRowRef(ParseState *pstate,
* "rel.*" mean the same thing for composite relations, so why not for
* scalar functions...
*/
- if (nsitem->p_names == nsitem->p_rte->eref)
+ if (nsitem->p_names == nsitem->p_rte->eref ||
+ nsitem->p_returning_type != VAR_RETURNING_DEFAULT)
{
Var *result;
result = makeWholeRowVar(nsitem->p_rte, nsitem->p_rtindex,
sublevels_up, true);
+ /* mark Var for RETURNING OLD/NEW, as necessary */
+ result->varreturningtype = nsitem->p_returning_type;
+
/* location is not filled in by makeWholeRowVar */
result->location = location;
@@ -2657,9 +2668,8 @@ transformWholeRowRef(ParseState *pstate,
* are in the RTE. We needn't worry about marking the RTE for SELECT
* access, as the common columns are surely so marked already.
*/
- expandRTE(nsitem->p_rte, nsitem->p_rtindex,
- sublevels_up, location, false,
- NULL, &fields);
+ expandRTE(nsitem->p_rte, nsitem->p_rtindex, sublevels_up,
+ nsitem->p_returning_type, location, false, NULL, &fields);
rowexpr = makeNode(RowExpr);
rowexpr->args = list_truncate(fields,
list_length(nsitem->p_names->colnames));
diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c
new file mode 100644
index 87df790..0eb8bb4
--- a/src/backend/parser/parse_merge.c
+++ b/src/backend/parser/parse_merge.c
@@ -247,8 +247,8 @@ transformMergeStmt(ParseState *pstate, M
qry->jointree = makeFromExpr(pstate->p_joinlist, NULL);
/* Transform the RETURNING list, if any */
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_MERGE_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_MERGE_RETURNING);
/*
* We now have a good query shape, so now look at the WHEN conditions and
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
new file mode 100644
index 8075b1b..610d879
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -91,11 +91,13 @@ static void markRTEForSelectPriv(ParseSt
int rtindex, AttrNumber col);
static void expandRelation(Oid relid, Alias *eref,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static int specialAttNum(const char *attname);
@@ -763,6 +765,9 @@ scanNSItemForColumn(ParseState *pstate,
}
var->location = location;
+ /* Mark Var for RETURNING OLD/NEW, as necessary */
+ var->varreturningtype = nsitem->p_returning_type;
+
/* Mark Var if it's nulled by any outer joins */
markNullableIfNeeded(pstate, var);
@@ -1336,6 +1341,7 @@ buildNSItemFromTupleDesc(RangeTblEntry *
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -1399,6 +1405,7 @@ buildNSItemFromLists(RangeTblEntry *rte,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2300,6 +2307,7 @@ addRangeTableEntryForJoin(ParseState *ps
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2720,9 +2728,10 @@ addNSItemToQuery(ParseState *pstate, Par
* results. If include_dropped is true then empty strings and NULL constants
* (not Vars!) are returned for dropped columns.
*
- * rtindex, sublevels_up, and location are the varno, varlevelsup, and location
- * values to use in the created Vars. Ordinarily rtindex should match the
- * actual position of the RTE in its rangetable.
+ * rtindex, sublevels_up, returning_type, and location are the varno,
+ * varlevelsup, varreturningtype, and location values to use in the created
+ * Vars. Ordinarily rtindex should match the actual position of the RTE in
+ * its rangetable.
*
* The output lists go into *colnames and *colvars.
* If only one of the two kinds of output list is needed, pass NULL for the
@@ -2730,6 +2739,7 @@ addNSItemToQuery(ParseState *pstate, Par
*/
void
expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -2745,7 +2755,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
case RTE_RELATION:
/* Ordinary relation RTE */
expandRelation(rte->relid, rte->eref,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up, returning_type, location,
include_dropped, colnames, colvars);
break;
case RTE_SUBQUERY:
@@ -2792,6 +2802,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
exprTypmod((Node *) te->expr),
exprCollation((Node *) te->expr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2829,7 +2840,8 @@ expandRTE(RangeTblEntry *rte, int rtinde
Assert(tupdesc);
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up,
+ returning_type, location,
include_dropped, colnames, colvars);
}
else if (functypclass == TYPEFUNC_SCALAR)
@@ -2849,6 +2861,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
exprTypmod(rtfunc->funcexpr),
exprCollation(rtfunc->funcexpr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2891,6 +2904,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
attrtypmod,
attrcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
}
@@ -2920,6 +2934,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
InvalidOid,
sublevels_up);
+ varnode->varreturningtype = returning_type;
*colvars = lappend(*colvars, varnode);
}
}
@@ -3002,6 +3017,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
exprTypmod(avar),
exprCollation(avar),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3057,6 +3073,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
varnode = makeVar(rtindex, varattno,
coltype, coltypmod, colcoll,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3089,6 +3106,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
*/
static void
expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3097,7 +3115,7 @@ expandRelation(Oid relid, Alias *eref, i
/* Get the tupledesc and turn it over to expandTupleDesc */
rel = relation_open(relid, AccessShareLock);
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
- rtindex, sublevels_up,
+ rtindex, sublevels_up, returning_type,
location, include_dropped,
colnames, colvars);
relation_close(rel, AccessShareLock);
@@ -3115,6 +3133,7 @@ expandRelation(Oid relid, Alias *eref, i
static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3175,6 +3194,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias
attr->atttypid, attr->atttypmod,
attr->attcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3227,6 +3247,7 @@ expandNSItemVars(ParseState *pstate, Par
nscol->p_varcollid,
sublevels_up);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
var->location = location;
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
new file mode 100644
index 76bf88c..f90afe2
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -1550,8 +1550,8 @@ expandRecordVariable(ParseState *pstate,
*lvar;
int i;
- expandRTE(rte, var->varno, 0, var->location, false,
- &names, &vars);
+ expandRTE(rte, var->varno, 0, var->varreturningtype,
+ var->location, false, &names, &vars);
tupleDesc = CreateTemplateTupleDesc(list_length(vars));
i = 1;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
new file mode 100644
index 6d59a2b..e8b86e2
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -635,6 +635,7 @@ rewriteRuleAction(Query *parsetree,
0,
rt_fetch(new_varno, sub_action->rtable),
parsetree->targetList,
+ sub_action->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -668,10 +669,15 @@ rewriteRuleAction(Query *parsetree,
rt_fetch(parsetree->resultRelation,
parsetree->rtable),
rule_action->returningList,
+ rule_action->resultRelation,
REPLACEVARS_REPORT_ERROR,
0,
&rule_action->hasSubLinks);
+ /* use triggering query's aliases for OLD and NEW in RETURNING list */
+ rule_action->returningOld = parsetree->returningOld;
+ rule_action->returningNew = parsetree->returningNew;
+
/*
* There could have been some SubLinks in parsetree's returningList,
* in which case we'd better mark the rule_action correctly.
@@ -2304,6 +2310,7 @@ CopyAndAddInvertedQual(Query *parsetree,
rt_fetch(rt_index,
parsetree->rtable),
parsetree->targetList,
+ parsetree->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -3528,6 +3535,7 @@ rewriteTargetView(Query *parsetree, Rela
0,
view_rte,
view_targetlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
NULL);
@@ -3679,6 +3687,7 @@ rewriteTargetView(Query *parsetree, Rela
0,
view_rte,
tmp_tlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
&parsetree->hasSubLinks);
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
new file mode 100644
index b20625f..fd74a4b
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -817,6 +817,14 @@ IncrementVarSublevelsUp_walker(Node *nod
phv->phlevelsup += context->delta_sublevels_up;
/* fall through to recurse into argument */
}
+ if (IsA(node, ReturningExpr))
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+
+ if (rexpr->retlevelsup >= context->min_sublevels_up)
+ rexpr->retlevelsup += context->delta_sublevels_up;
+ /* fall through to recurse into argument */
+ }
if (IsA(node, RangeTblEntry))
{
RangeTblEntry *rte = (RangeTblEntry *) node;
@@ -883,6 +891,68 @@ IncrementVarSublevelsUp_rtable(List *rta
QTW_EXAMINE_RTES_BEFORE);
}
+/*
+ * SetVarReturningType - adjust Var nodes for a specified varreturningtype.
+ *
+ * Find all Var nodes referring to the specified result relation in the given
+ * expression and set their varreturningtype to the specified value.
+ *
+ * NOTE: although this has the form of a walker, we cheat and modify the
+ * Var nodes in-place. The given expression tree should have been copied
+ * earlier to ensure that no unwanted side-effects occur!
+ */
+
+typedef struct
+{
+ int result_relation;
+ int sublevels_up;
+ VarReturningType returning_type;
+} SetVarReturningType_context;
+
+static bool
+SetVarReturningType_walker(Node *node, SetVarReturningType_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ Var *var = (Var *) node;
+
+ if (var->varno == context->result_relation &&
+ var->varlevelsup == context->sublevels_up)
+ var->varreturningtype = context->returning_type;
+
+ return false;
+ }
+
+ if (IsA(node, Query))
+ {
+ /* Recurse into subselects */
+ bool result;
+
+ context->sublevels_up++;
+ result = query_tree_walker((Query *) node, SetVarReturningType_walker,
+ (void *) context, 0);
+ context->sublevels_up--;
+ return result;
+ }
+ return expression_tree_walker(node, SetVarReturningType_walker,
+ (void *) context);
+}
+
+static void
+SetVarReturningType(Node *node, int result_relation, int sublevels_up,
+ VarReturningType returning_type)
+{
+ SetVarReturningType_context context;
+
+ context.result_relation = result_relation;
+ context.sublevels_up = sublevels_up;
+ context.returning_type = returning_type;
+
+ /* Expect to start with an expression */
+ SetVarReturningType_walker(node, &context);
+}
/*
* rangeTableEntry_used - detect whether an RTE is referenced somewhere
@@ -1653,6 +1723,15 @@ map_variable_attnos(Node *node,
* relation. This is needed to handle whole-row Vars referencing the target.
* We expand such Vars into RowExpr constructs.
*
+ * In addition, for INSERT/UPDATE/DELETE/MERGE queries, the caller must
+ * provide result_relation, the index of the result relation in the rewritten
+ * query. This is needed to handle OLD/NEW RETURNING list Vars referencing
+ * target_varno. When such Vars are expanded, their varreturningtype is
+ * copied onto any replacement Vars referencing result_relation. In addition,
+ * if the replacement expression from the targetlist is not simply a Var
+ * referencing result_relation, it is wrapped in a ReturningExpr node (causing
+ * the executor to return NULL if the OLD/NEW row doesn't exist).
+ *
* outer_hasSubLinks works the same as for replace_rte_variables().
*/
@@ -1660,6 +1739,7 @@ typedef struct
{
RangeTblEntry *target_rte;
List *targetlist;
+ int result_relation;
ReplaceVarsNoMatchOption nomatch_option;
int nomatch_varno;
} ReplaceVarsFromTargetList_context;
@@ -1684,10 +1764,13 @@ ReplaceVarsFromTargetList_callback(Var *
* dropped columns. If the var is RECORD (ie, this is a JOIN), then
* omit dropped columns. In the latter case, attach column names to
* the RowExpr for use of the executor and ruleutils.c.
+ *
+ * The varreturningtype is copied onto each individual field Var, so
+ * that it is handled correctly when we recurse.
*/
expandRTE(rcon->target_rte,
- var->varno, var->varlevelsup, var->location,
- (var->vartype != RECORDOID),
+ var->varno, var->varlevelsup, var->varreturningtype,
+ var->location, (var->vartype != RECORDOID),
&colnames, &fields);
/* Adjust the generated per-field Vars... */
fields = (List *) replace_rte_variables_mutator((Node *) fields,
@@ -1699,6 +1782,18 @@ ReplaceVarsFromTargetList_callback(Var *
rowexpr->colnames = (var->vartype == RECORDOID) ? colnames : NIL;
rowexpr->location = var->location;
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = var->varreturningtype == VAR_RETURNING_OLD;
+ rexpr->retexpr = (Expr *) rowexpr;
+
+ return (Node *) rexpr;
+ }
+
return (Node *) rowexpr;
}
@@ -1764,6 +1859,34 @@ ReplaceVarsFromTargetList_callback(Var *
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("NEW variables in ON UPDATE rules cannot reference columns that are part of a multiple assignment in the subject UPDATE command")));
+ /* Handle any OLD/NEW RETURNING list Vars */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ /*
+ * Copy varreturningtype onto any Vars in the tlist item that
+ * refer to result_relation (which had better be non-zero).
+ */
+ if (rcon->result_relation == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+
+ SetVarReturningType((Node *) newnode, rcon->result_relation,
+ var->varlevelsup, var->varreturningtype);
+
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (!IsA(newnode, Var) ||
+ ((Var *) newnode)->varno != rcon->result_relation ||
+ ((Var *) newnode)->varlevelsup != var->varlevelsup)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = (var->varreturningtype == VAR_RETURNING_OLD);
+ rexpr->retexpr = newnode;
+
+ newnode = (Expr *) rexpr;
+ }
+ }
+
return (Node *) newnode;
}
}
@@ -1773,6 +1896,7 @@ ReplaceVarsFromTargetList(Node *node,
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks)
@@ -1781,6 +1905,7 @@ ReplaceVarsFromTargetList(Node *node,
context.target_rte = target_rte;
context.targetlist = targetlist;
+ context.result_relation = result_relation;
context.nomatch_option = nomatch_option;
context.nomatch_varno = nomatch_varno;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
new file mode 100644
index 2177d17..bbef920
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -167,6 +167,8 @@ typedef struct
List *subplans; /* List of Plan trees for SubPlans */
List *ctes; /* List of CommonTableExpr nodes */
AppendRelInfo **appendrels; /* Array of AppendRelInfo nodes, or NULL */
+ char *returningOld; /* alias for OLD in RETURNING list */
+ char *returningNew; /* alias for NEW in RETURNING list */
/* Workspace for column alias assignment: */
bool unique_using; /* Are we making USING names globally unique */
List *using_names; /* List of assigned names for USING columns */
@@ -426,6 +428,7 @@ static void get_merge_query_def(Query *q
static void get_utility_query_def(Query *query, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context);
static void get_target_list(List *targetList, deparse_context *context);
+static void get_returning_clause(Query *query, deparse_context *context);
static void get_setop_query(Node *setOp, Query *query,
deparse_context *context);
static Node *get_rule_sortgroupclause(Index ref, List *tlist,
@@ -3779,6 +3782,10 @@ deparse_context_for_plan_tree(PlannedStm
* the most-closely-nested first. This is needed to resolve PARAM_EXEC
* Params. Note we assume that all the Plan nodes share the same rtable.
*
+ * For a ModifyTable plan, we might also need to resolve references to OLD/NEW
+ * variables in the RETURNING list, so we copy the alias names of the OLD and
+ * NEW rows from the ModifyTable plan node.
+ *
* Once this function has been called, deparse_expression() can be called on
* subsidiary expression(s) of the specified Plan node. To deparse
* expressions of a different Plan node in the same Plan tree, re-call this
@@ -3799,6 +3806,13 @@ set_deparse_context_plan(List *dpcontext
dpns->ancestors = ancestors;
set_deparse_plan(dpns, plan);
+ /* For ModifyTable, set aliases for OLD and NEW in RETURNING */
+ if (IsA(plan, ModifyTable))
+ {
+ dpns->returningOld = ((ModifyTable *) plan)->returningOld;
+ dpns->returningNew = ((ModifyTable *) plan)->returningNew;
+ }
+
return dpcontext;
}
@@ -3996,6 +4010,8 @@ set_deparse_for_query(deparse_namespace
dpns->subplans = NIL;
dpns->ctes = query->cteList;
dpns->appendrels = NULL;
+ dpns->returningOld = query->returningOld;
+ dpns->returningNew = query->returningNew;
/* Assign a unique relation alias to each RTE */
set_rtable_names(dpns, parent_namespaces, NULL);
@@ -4387,8 +4403,8 @@ set_relation_column_names(deparse_namesp
if (rte->rtekind == RTE_FUNCTION && rte->functions != NIL)
{
/* Since we're not creating Vars, rtindex etc. don't matter */
- expandRTE(rte, 1, 0, -1, true /* include dropped */ ,
- &colnames, NULL);
+ expandRTE(rte, 1, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , &colnames, NULL);
}
else
colnames = rte->eref->colnames;
@@ -6315,6 +6331,43 @@ get_target_list(List *targetList, depars
}
static void
+get_returning_clause(Query *query, deparse_context *context)
+{
+ StringInfo buf = context->buf;
+
+ if (query->returningList)
+ {
+ bool have_with = false;
+
+ appendContextKeyword(context, " RETURNING",
+ -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
+
+ /* Add WITH options, if they're not the defaults */
+ if (query->returningOld && strcmp(query->returningOld, "old") != 0)
+ {
+ appendStringInfo(buf, " WITH (OLD AS %s", query->returningOld);
+ have_with = true;
+ }
+ if (query->returningNew && strcmp(query->returningNew, "new") != 0)
+ {
+ if (have_with)
+ appendStringInfoString(buf, ", ");
+ else
+ {
+ appendStringInfoString(buf, " WITH (");
+ have_with = true;
+ }
+ appendStringInfo(buf, "NEW AS %s", query->returningNew);
+ }
+ if (have_with)
+ appendStringInfoChar(buf, ')');
+
+ /* Add the returning expressions themselves */
+ get_target_list(query->returningList, context);
+ }
+}
+
+static void
get_setop_query(Node *setOp, Query *query, deparse_context *context)
{
StringInfo buf = context->buf;
@@ -6988,11 +7041,7 @@ get_insert_query_def(Query *query, depar
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7044,11 +7093,7 @@ get_update_query_def(Query *query, depar
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7247,11 +7292,7 @@ get_delete_query_def(Query *query, depar
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7410,11 +7451,7 @@ get_merge_query_def(Query *query, depars
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7562,7 +7599,13 @@ get_variable(Var *var, int levelsup, boo
}
rte = rt_fetch(varno, dpns->rtable);
- refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+ if (var->varreturningtype == VAR_RETURNING_OLD)
+ refname = dpns->returningOld;
+ else if (var->varreturningtype == VAR_RETURNING_NEW)
+ refname = dpns->returningNew;
+ else
+ refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+
colinfo = deparse_columns_fetch(varno, dpns);
attnum = varattno;
}
@@ -7676,7 +7719,8 @@ get_variable(Var *var, int levelsup, boo
attname = get_rte_attribute_name(rte, attnum);
}
- need_prefix = (context->varprefix || attname == NULL);
+ need_prefix = (context->varprefix || attname == NULL ||
+ var->varreturningtype != VAR_RETURNING_DEFAULT);
/*
* If we're considering a plain Var in an ORDER BY (but not GROUP BY)
@@ -8727,6 +8771,7 @@ isSimpleNode(Node *node, Node *parentNod
case T_SQLValueFunction:
case T_XmlExpr:
case T_NextValueExpr:
+ case T_ReturningExpr:
case T_NullIfExpr:
case T_Aggref:
case T_GroupingFunc:
@@ -8849,6 +8894,7 @@ isSimpleNode(Node *node, Node *parentNod
case T_CoalesceExpr: /* own parentheses */
case T_MinMaxExpr: /* own parentheses */
case T_XmlExpr: /* own parentheses */
+ case T_ReturningExpr: /* own parentheses */
case T_NullIfExpr: /* other separators */
case T_Aggref: /* own parentheses */
case T_GroupingFunc: /* own parentheses */
@@ -8901,6 +8947,7 @@ isSimpleNode(Node *node, Node *parentNod
case T_CoalesceExpr: /* own parentheses */
case T_MinMaxExpr: /* own parentheses */
case T_XmlExpr: /* own parentheses */
+ case T_ReturningExpr: /* own parentheses */
case T_NullIfExpr: /* other separators */
case T_Aggref: /* own parentheses */
case T_GroupingFunc: /* own parentheses */
@@ -10258,6 +10305,17 @@ get_rule_expr(Node *node, deparse_contex
}
break;
+ case T_ReturningExpr:
+ /* Returns old/new.(expression) */
+ if (((ReturningExpr *) node)->retold)
+ appendStringInfoString(buf, "old.(");
+ else
+ appendStringInfoString(buf, "new.(");
+ get_rule_expr((Node *) ((ReturningExpr *) node)->retexpr,
+ context, showimplicit);
+ appendStringInfoChar(buf, ')');
+ break;
+
case T_PartitionBoundSpec:
{
PartitionBoundSpec *spec = (PartitionBoundSpec *) node;
diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h
new file mode 100644
index eec0aa6..27dd70d
--- a/src/include/executor/execExpr.h
+++ b/src/include/executor/execExpr.h
@@ -26,9 +26,9 @@ struct JsonConstructorExprState;
/* Bits in ExprState->flags (see also execnodes.h for public flag bits): */
/* expression's interpreter has been initialized */
-#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 1)
+#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 5)
/* jump-threading is in use */
-#define EEO_FLAG_DIRECT_THREADED (1 << 2)
+#define EEO_FLAG_DIRECT_THREADED (1 << 6)
/* Typical API for out-of-line evaluation subroutines */
typedef void (*ExecEvalSubroutine) (ExprState *state,
@@ -72,16 +72,22 @@ typedef enum ExprEvalOp
EEOP_INNER_FETCHSOME,
EEOP_OUTER_FETCHSOME,
EEOP_SCAN_FETCHSOME,
+ EEOP_OLD_FETCHSOME,
+ EEOP_NEW_FETCHSOME,
/* compute non-system Var value */
EEOP_INNER_VAR,
EEOP_OUTER_VAR,
EEOP_SCAN_VAR,
+ EEOP_OLD_VAR,
+ EEOP_NEW_VAR,
/* compute system Var value */
EEOP_INNER_SYSVAR,
EEOP_OUTER_SYSVAR,
EEOP_SCAN_SYSVAR,
+ EEOP_OLD_SYSVAR,
+ EEOP_NEW_SYSVAR,
/* compute wholerow Var */
EEOP_WHOLEROW,
@@ -94,6 +100,8 @@ typedef enum ExprEvalOp
EEOP_ASSIGN_INNER_VAR,
EEOP_ASSIGN_OUTER_VAR,
EEOP_ASSIGN_SCAN_VAR,
+ EEOP_ASSIGN_OLD_VAR,
+ EEOP_ASSIGN_NEW_VAR,
/* assign ExprState's resvalue/resnull to a column of its resultslot */
EEOP_ASSIGN_TMP,
@@ -178,6 +186,7 @@ typedef enum ExprEvalOp
EEOP_SQLVALUEFUNCTION,
EEOP_CURRENTOFEXPR,
EEOP_NEXTVALUEEXPR,
+ EEOP_RETURNINGEXPR,
EEOP_ARRAYEXPR,
EEOP_ARRAYCOERCE,
EEOP_ROW,
@@ -301,7 +310,7 @@ typedef struct ExprEvalStep
*/
union
{
- /* for EEOP_INNER/OUTER/SCAN_FETCHSOME */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_FETCHSOME */
struct
{
/* attribute number up to which to fetch (inclusive) */
@@ -314,13 +323,14 @@ typedef struct ExprEvalStep
const TupleTableSlotOps *kind;
} fetch;
- /* for EEOP_INNER/OUTER/SCAN_[SYS]VAR[_FIRST] */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_[SYS]VAR */
struct
{
/* attnum is attr number - 1 for regular VAR ... */
/* but it's just the normal (negative) attr number for SYSVAR */
int attnum;
Oid vartype; /* type OID of variable */
+ VarReturningType varreturningtype; /* return old/new/default */
} var;
/* for EEOP_WHOLEROW */
@@ -349,6 +359,13 @@ typedef struct ExprEvalStep
int resultnum;
} assign_tmp;
+ /* for EEOP_RETURNINGEXPR */
+ struct
+ {
+ uint8 nullflag; /* flag to test if OLD/NEW row is NULL */
+ int jumpdone; /* jump here if OLD/NEW row is NULL */
+ } returningexpr;
+
/* for EEOP_CONST */
struct
{
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
new file mode 100644
index 69c3ebf..ea1eed1
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -624,6 +624,7 @@ extern int ExecCleanTargetListLength(Lis
extern TupleTableSlot *ExecGetTriggerOldSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetTriggerNewSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetReturningSlot(EState *estate, ResultRelInfo *relInfo);
+extern TupleTableSlot *ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleConversionMap *ExecGetChildToRootMap(ResultRelInfo *resultRelInfo);
extern TupleConversionMap *ExecGetRootToChildMap(ResultRelInfo *resultRelInfo, EState *estate);
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
new file mode 100644
index e4698a2..5746917
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -74,11 +74,20 @@ typedef Datum (*ExprStateEvalFunc) (stru
/* Bits in ExprState->flags (see also execExpr.h for private flag bits): */
/* expression is for use with ExecQual() */
#define EEO_FLAG_IS_QUAL (1 << 0)
+/* expression refers to OLD table columns */
+#define EEO_FLAG_HAS_OLD (1 << 1)
+/* expression refers to NEW table columns */
+#define EEO_FLAG_HAS_NEW (1 << 2)
+/* OLD table row is NULL in RETURNING list */
+#define EEO_FLAG_OLD_IS_NULL (1 << 3)
+/* NEW table row is NULL in RETURNING list */
+#define EEO_FLAG_NEW_IS_NULL (1 << 4)
typedef struct ExprState
{
NodeTag type;
+#define FIELDNO_EXPRSTATE_FLAGS 1
uint8 flags; /* bitmask of EEO_FLAG_* bits, see above */
/*
@@ -290,6 +299,12 @@ typedef struct ExprContext
#define FIELDNO_EXPRCONTEXT_DOMAINNULL 13
bool domainValue_isNull;
+ /* Tuples that OLD/NEW Var nodes in RETURNING may refer to */
+#define FIELDNO_EXPRCONTEXT_OLDTUPLE 14
+ TupleTableSlot *ecxt_oldtuple;
+#define FIELDNO_EXPRCONTEXT_NEWTUPLE 15
+ TupleTableSlot *ecxt_newtuple;
+
/* Link to containing EState (NULL if a standalone ExprContext) */
struct EState *ecxt_estate;
@@ -504,6 +519,7 @@ typedef struct ResultRelInfo
TupleTableSlot *ri_ReturningSlot; /* for trigger output tuples */
TupleTableSlot *ri_TrigOldSlot; /* for a trigger's old tuple */
TupleTableSlot *ri_TrigNewSlot; /* for a trigger's new tuple */
+ TupleTableSlot *ri_AllNullSlot; /* for RETURNING OLD/NEW */
/* FDW callback functions, if foreign table */
struct FdwRoutine *ri_FdwRoutine;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
new file mode 100644
index 5b62df3..848610a
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -197,6 +197,16 @@ typedef struct Query
OnConflictExpr *onConflict; /* ON CONFLICT DO [NOTHING | UPDATE] */
+ /*
+ * The following three fields describe the contents of the RETURNING list
+ * for INSERT/UPDATE/DELETE/MERGE. If returningOld or returningNew are
+ * non-NULL, then returningList may contain entries referring to old/new
+ * values in the result relation; if they are NULL, the default old/new
+ * alias was masked by a user-supplied alias/table name, and returningList
+ * cannot return old/new values.
+ */
+ char *returningOld; /* alias for OLD in RETURNING list */
+ char *returningNew; /* alias for NEW in RETURNING list */
List *returningList; /* return-values list (of TargetEntry) */
List *groupClause; /* a list of SortGroupClause's */
@@ -1726,6 +1736,41 @@ typedef struct MergeWhenClause
} MergeWhenClause;
/*
+ * ReturningOptionKind -
+ * Possible kinds of option in RETURNING WITH(...) list
+ *
+ * Currently, this is used only for specifying OLD/NEW aliases.
+ */
+typedef enum ReturningOptionKind
+{
+ RETURNING_OPTION_OLD, /* specify alias for OLD in RETURNING */
+ RETURNING_OPTION_NEW, /* specify alias for NEW in RETURNING */
+} ReturningOptionKind;
+
+/*
+ * ReturningOption -
+ * An individual option in the RETURNING WITH(...) list
+ */
+typedef struct ReturningOption
+{
+ NodeTag type;
+ ReturningOptionKind option; /* specified option */
+ char *value; /* option's value */
+ ParseLoc location; /* token location, or -1 if unknown */
+} ReturningOption;
+
+/*
+ * ReturningClause -
+ * List of RETURNING expressions, together with any WITH(...) options
+ */
+typedef struct ReturningClause
+{
+ NodeTag type;
+ List *options; /* list of ReturningOption elements */
+ List *exprs; /* list of expressions to return */
+} ReturningClause;
+
+/*
* TriggerTransition -
* representation of transition row or table naming clause
*
@@ -2042,7 +2087,7 @@ typedef struct InsertStmt
List *cols; /* optional: names of the target columns */
Node *selectStmt; /* the source SELECT/VALUES, or NULL */
OnConflictClause *onConflictClause; /* ON CONFLICT clause */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
OverridingKind override; /* OVERRIDING clause */
} InsertStmt;
@@ -2057,7 +2102,7 @@ typedef struct DeleteStmt
RangeVar *relation; /* relation to delete from */
List *usingClause; /* optional using clause for more tables */
Node *whereClause; /* qualifications */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
} DeleteStmt;
@@ -2072,7 +2117,7 @@ typedef struct UpdateStmt
List *targetList; /* the target list (of ResTarget) */
Node *whereClause; /* qualifications */
List *fromClause; /* optional from clause for more tables */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
} UpdateStmt;
@@ -2087,7 +2132,7 @@ typedef struct MergeStmt
Node *sourceRelation; /* source relation */
Node *joinCondition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
} MergeStmt;
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
new file mode 100644
index 62cd6a6..c37d421
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -238,6 +238,8 @@ typedef struct ModifyTable
List *resultRelations; /* integer list of RT indexes */
List *updateColnosLists; /* per-target-table update_colnos lists */
List *withCheckOptionLists; /* per-target-table WCO lists */
+ char *returningOld; /* alias for OLD in RETURNING lists */
+ char *returningNew; /* alias for NEW in RETURNING lists */
List *returningLists; /* per-target-table RETURNING tlists */
List *fdwPrivLists; /* per-target-table FDW private data lists */
Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
new file mode 100644
index ea47652..1060fcf
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -223,6 +223,12 @@ typedef struct Expr
* Note that it affects the meaning of all of varno, varnullingrels, and
* varnosyn, all of which refer to the range table of that query level.
*
+ * varreturningtype is used for Vars that refer to the target relation in the
+ * RETURNING list of data-modifying queries. The default behavior is to
+ * return old values for DELETE operations and new values for INSERT and
+ * UPDATE operations, but it is also possible to explicitly request old/new
+ * values by referring to the target relation using the OLD/NEW aliases.
+ *
* In the parser, varnosyn and varattnosyn are either identical to
* varno/varattno, or they specify the column's position in an aliased JOIN
* RTE that hides the semantic referent RTE's refname. This is a syntactic
@@ -244,6 +250,14 @@ typedef struct Expr
#define PRS2_OLD_VARNO 1
#define PRS2_NEW_VARNO 2
+/* Returning behavior for Vars in RETURNING list */
+typedef enum VarReturningType
+{
+ VAR_RETURNING_DEFAULT, /* return OLD for DELETE, else return NEW */
+ VAR_RETURNING_OLD, /* return OLD for DELETE/UPDATE, else NULL */
+ VAR_RETURNING_NEW, /* return NEW for INSERT/UPDATE, else NULL */
+} VarReturningType;
+
typedef struct Var
{
Expr xpr;
@@ -279,6 +293,9 @@ typedef struct Var
*/
Index varlevelsup;
+ /* returning type of this var (see above) */
+ VarReturningType varreturningtype;
+
/*
* varnosyn/varattnosyn are ignored for equality, because Vars with
* different syntactic identifiers are semantically the same as long as
@@ -2124,6 +2141,30 @@ typedef struct InferenceElem
Oid inferopclass; /* OID of att opclass, or InvalidOid */
} InferenceElem;
+/*
+ * ReturningExpr - return OLD/NEW.(expression) in RETURNING list
+ *
+ * This is used when updating an auto-updatable view and returning a view
+ * column that is not simply a Var referring to the base relation. In such
+ * cases, OLD/NEW.viewcol can expand to an arbitrary expression, but the
+ * result is required to be NULL if the OLD/NEW row doesn't exist. To handle
+ * this, the rewriter wraps the expanded expression in a ReturningExpr, which
+ * is equivalent to "CASE WHEN (OLD/NEW row exists) THEN (expr) ELSE NULL".
+ *
+ * A similar situation can arise when rewriting the RETURNING clause of a
+ * rule, which may also contain arbitrary expressions.
+ *
+ * ReturningExpr nodes never appear in a parsed Query --- they are only ever
+ * inserted by the rewriter.
+ */
+typedef struct ReturningExpr
+{
+ Expr xpr;
+ int retlevelsup; /* > 0 if it belongs to outer query */
+ bool retold; /* true for OLD, false for NEW */
+ Expr *retexpr; /* expression to be returned */
+} ReturningExpr;
+
/*--------------------
* TargetEntry -
* a target entry (used in query target lists)
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
new file mode 100644
index 93e3dc7..a6ab887
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -198,6 +198,7 @@ extern void pull_varattnos(Node *node, I
extern List *pull_vars_of_level(Node *node, int levelsup);
extern bool contain_var_clause(Node *node);
extern bool contain_vars_of_level(Node *node, int levelsup);
+extern bool contain_vars_returning_old_or_new(Node *node);
extern int locate_var_of_level(Node *node, int levelsup);
extern List *pull_var_clause(Node *node, int flags);
extern Node *flatten_join_alias_vars(PlannerInfo *root, Query *query, Node *node);
diff --git a/src/include/optimizer/paramassign.h b/src/include/optimizer/paramassign.h
new file mode 100644
index 4026b74..89d2d07
--- a/src/include/optimizer/paramassign.h
+++ b/src/include/optimizer/paramassign.h
@@ -22,6 +22,8 @@ extern Param *replace_outer_agg(PlannerI
extern Param *replace_outer_grouping(PlannerInfo *root, GroupingFunc *grp);
extern Param *replace_outer_merge_support(PlannerInfo *root,
MergeSupportFunc *msf);
+extern Param *replace_outer_returning(PlannerInfo *root,
+ ReturningExpr *rexpr);
extern Param *replace_nestloop_param_var(PlannerInfo *root, Var *var);
extern Param *replace_nestloop_param_placeholdervar(PlannerInfo *root,
PlaceHolderVar *phv);
diff --git a/src/include/parser/analyze.h b/src/include/parser/analyze.h
new file mode 100644
index 28b66fc..37f3bd3
--- a/src/include/parser/analyze.h
+++ b/src/include/parser/analyze.h
@@ -44,8 +44,9 @@ extern List *transformInsertRow(ParseSta
bool strip_indirection);
extern List *transformUpdateTargetList(ParseState *pstate,
List *origTlist);
-extern List *transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind);
+extern void transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind);
extern Query *transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree);
extern Query *transformStmt(ParseState *pstate, Node *parseTree);
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
new file mode 100644
index 543df56..301fa42
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -279,6 +279,11 @@ struct ParseState
* of SQL:2008 requires us to do it this way. We also use p_lateral_ok to
* forbid LATERAL references to an UPDATE/DELETE target table.
*
+ * While processing the RETURNING clause, special namespace items are added to
+ * refer to the OLD and NEW state of the result relation. These namespace
+ * items have p_returning_type set appropriately, for use when creating Vars.
+ * For convenience, this information is duplicated on each namespace column.
+ *
* At no time should a namespace list contain two entries that conflict
* according to the rules in checkNameSpaceConflicts; but note that those
* are more complicated than "must have different alias names", so in practice
@@ -296,6 +301,7 @@ struct ParseNamespaceItem
bool p_cols_visible; /* Column names visible as unqualified refs? */
bool p_lateral_only; /* Is only visible to LATERAL expressions? */
bool p_lateral_ok; /* If so, does join type allow use? */
+ VarReturningType p_returning_type; /* Is OLD/NEW for use in RETURNING? */
};
/*
@@ -326,6 +332,7 @@ struct ParseNamespaceColumn
Oid p_vartype; /* pg_type OID */
int32 p_vartypmod; /* type modifier value */
Oid p_varcollid; /* OID of collation, or InvalidOid */
+ VarReturningType p_varreturningtype; /* for RETURNING OLD/NEW */
Index p_varnosyn; /* rangetable index of syntactic referent */
AttrNumber p_varattnosyn; /* attribute number of syntactic referent */
bool p_dontexpand; /* not included in star expansion */
diff --git a/src/include/parser/parse_relation.h b/src/include/parser/parse_relation.h
new file mode 100644
index 91fd8e2..3dcc1ab
--- a/src/include/parser/parse_relation.h
+++ b/src/include/parser/parse_relation.h
@@ -114,6 +114,7 @@ extern void errorMissingRTE(ParseState *
extern void errorMissingColumn(ParseState *pstate,
const char *relname, const char *colname, int location) pg_attribute_noreturn();
extern void expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
extern List *expandNSItemVars(ParseState *pstate, ParseNamespaceItem *nsitem,
diff --git a/src/include/rewrite/rewriteManip.h b/src/include/rewrite/rewriteManip.h
new file mode 100644
index ac6d204..15839ac
--- a/src/include/rewrite/rewriteManip.h
+++ b/src/include/rewrite/rewriteManip.h
@@ -89,6 +89,7 @@ extern Node *ReplaceVarsFromTargetList(N
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks);
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
new file mode 100644
index fe8d3e5..a7420ff
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -119,8 +119,8 @@ my %replace_line = (
'SHOW TRANSACTION ISOLATION LEVEL ecpg_into',
'VariableShowStmtSHOWSESSIONAUTHORIZATION' =>
'SHOW SESSION AUTHORIZATION ecpg_into',
- 'returning_clauseRETURNINGtarget_list' =>
- 'RETURNING target_list opt_ecpg_into',
+ 'returning_clauseRETURNINGreturning_with_clausetarget_list' =>
+ 'RETURNING returning_with_clause target_list opt_ecpg_into',
'ExecuteStmtEXECUTEnameexecute_param_clause' =>
'EXECUTE prepared_name execute_param_clause execute_rest',
'ExecuteStmtCREATEOptTempTABLEcreate_as_targetASEXECUTEnameexecute_param_clauseopt_with_data'
diff --git a/src/test/isolation/expected/merge-update.out b/src/test/isolation/expected/merge-update.out
new file mode 100644
index 3063c0c..677263d
--- a/src/test/isolation/expected/merge-update.out
+++ b/src/test/isolation/expected/merge-update.out
@@ -40,12 +40,12 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -98,14 +98,14 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -137,13 +137,13 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step a1: ABORT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+-------------------------
-UPDATE | 2|setup1 updated by merge2a
+merge_action|old |new |key|val
+------------+----------+-------------------------------+---+-------------------------
+UPDATE |(1,setup1)|(2,"setup1 updated by merge2a")| 2|setup1 updated by merge2a
(1 row)
step select2: SELECT * FROM target;
@@ -234,14 +234,14 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
-merge_action|key|val
-------------+---+--------------------------------------------------
-UPDATE | 2|initial updated by pa_merge1 updated by pa_merge2a
-UPDATE | 3|initial source not matched by pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+--------------------------------------------------------+---+--------------------------------------------------
+UPDATE |(1,"initial updated by pa_merge1")|(2,"initial updated by pa_merge1 updated by pa_merge2a")| 2|initial updated by pa_merge1 updated by pa_merge2a
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
(2 rows)
step pa_select2: SELECT * FROM pa_target;
@@ -273,7 +273,7 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
@@ -303,13 +303,13 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+-------------------------------------------------------------
-UPDATE | 3|initial source not matched by pa_merge2a
-UPDATE | 3|initial updated by pa_merge2 source not matched by pa_merge2a
-INSERT | 1|pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+-------------------------------------------------------------------+---+-------------------------------------------------------------
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
+UPDATE |(2,"initial updated by pa_merge2")|(3,"initial updated by pa_merge2 source not matched by pa_merge2a")| 3|initial updated by pa_merge2 source not matched by pa_merge2a
+INSERT | |(1,pa_merge2a) | 1|pa_merge2a
(3 rows)
step pa_select2: SELECT * FROM pa_target;
diff --git a/src/test/isolation/specs/merge-update.spec b/src/test/isolation/specs/merge-update.spec
new file mode 100644
index a33dcdb..c718ff6
--- a/src/test/isolation/specs/merge-update.spec
+++ b/src/test/isolation/specs/merge-update.spec
@@ -95,7 +95,7 @@ step "merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
step "merge2b"
{
@@ -128,7 +128,7 @@ step "pa_merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
# MERGE proceeds only if 'val' unchanged
step "pa_merge2b_when"
diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out
new file mode 100644
index c236f15..ee774a1
--- a/src/test/regress/expected/merge.out
+++ b/src/test/regress/expected/merge.out
@@ -297,13 +297,13 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
- merge_action | tid | balance
---------------+-----+---------
- DELETE | 1 | 10
- DELETE | 2 | 20
- DELETE | 3 | 30
- INSERT | 4 | 40
+RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ DELETE | (1,10) | | 1 | 10
+ DELETE | (2,20) | | 2 | 20
+ DELETE | (3,30) | | 3 | 30
+ INSERT | | (4,40) | 4 | 40
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -994,7 +994,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 T
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
NOTICE: BEFORE INSERT STATEMENT trigger
NOTICE: BEFORE UPDATE STATEMENT trigger
NOTICE: BEFORE DELETE STATEMENT trigger
@@ -1009,12 +1009,12 @@ NOTICE: AFTER UPDATE ROW trigger row: (
NOTICE: AFTER DELETE STATEMENT trigger
NOTICE: AFTER UPDATE STATEMENT trigger
NOTICE: AFTER INSERT STATEMENT trigger
- merge_action | tid | balance
---------------+-----+---------
- UPDATE | 3 | 10
- INSERT | 4 | 40
- DELETE | 2 | 20
- UPDATE | 1 | 0
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ UPDATE | (3,30) | (3,10) | 3 | 10
+ INSERT | | (4,40) | 4 | 40
+ DELETE | (2,20) | | 2 | 20
+ UPDATE | (1,10) | (1,0) | 1 | 0
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -1436,17 +1436,19 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
WHEN 'DELETE' THEN 'Removed '||t
END AS description;
- action | tid | balance | description
---------+-----+---------+---------------------
- del | 1 | 100 | Removed (1,100)
- upd | 2 | 220 | Added 20 to balance
- ins | 4 | 40 | Inserted (4,40)
+ action | old_tid | old_balance | new_tid | new_balance | delta_balance | tid | balance | description
+--------+---------+-------------+---------+-------------+---------------+-----+---------+---------------------
+ del | 1 | 100 | | | | 1 | 100 | Removed (1,100)
+ upd | 2 | 200 | 2 | 220 | 20 | 2 | 220 | Added 20 to balance
+ ins | | | 4 | 40 | | 4 | 40 | Inserted (4,40)
(3 rows)
ROLLBACK;
@@ -1473,7 +1475,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -1487,14 +1489,14 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
- action | log_action | tid | last_change
---------+------------+-----+---------------------
- DELETE | UPDATE | 1 | Removed (1,100)
- UPDATE | INSERT | 2 | Added 20 to balance
- INSERT | INSERT | 4 | Inserted (4,40)
+ action | old_data | new_data | tid | balance | description | log_action | old_log | new_log | tid | last_change
+--------+----------+----------+-----+---------+---------------------+------------+----------------------+---------------------------+-----+---------------------
+ DELETE | (1,100) | | 1 | 100 | Removed (1,100) | UPDATE | (1,"Original value") | (1,"Removed (1,100)") | 1 | Removed (1,100)
+ UPDATE | (2,200) | (2,220) | 2 | 220 | Added 20 to balance | INSERT | | (2,"Added 20 to balance") | 2 | Added 20 to balance
+ INSERT | | (4,40) | 4 | 40 | Inserted (4,40) | INSERT | | (4,"Inserted (4,40)") | 4 | Inserted (4,40)
(3 rows)
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -1518,11 +1520,11 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
-DELETE 1 100
-UPDATE 2 220
-INSERT 4 40
+DELETE 1 100 \N \N
+UPDATE 2 200 2 220
+INSERT \N \N 4 40
ROLLBACK;
-- SQL function with MERGE ... RETURNING
BEGIN;
@@ -2039,10 +2041,10 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
- merge_action | tid | balance | val
---------------+-----+---------+--------------------------
- UPDATE | 2 | 110 | initial updated by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance | val
+--------------+-----------------+------------------------------------+-----+---------+--------------------------
+ UPDATE | (1,100,initial) | (2,110,"initial updated by merge") | 2 | 110 | initial updated by merge
(1 row)
SELECT * FROM pa_target ORDER BY tid;
@@ -2324,18 +2326,18 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
- merge_action | logts | tid | balance | val
---------------+--------------------------+-----+---------+--------------------------
- UPDATE | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | logts | tid | balance | val
+--------------+--------------------------------------------+---------------------------------------------------------------+--------------------------+-----+---------+--------------------------
+ UPDATE | ("Tue Jan 31 00:00:00 2017",1,100,initial) | ("Tue Jan 31 00:00:00 2017",1,110,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",2,200,initial) | ("Tue Feb 28 00:00:00 2017",2,220,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",3,30,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",4,400,initial) | ("Tue Jan 31 00:00:00 2017",4,440,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",5,500,initial) | ("Tue Feb 28 00:00:00 2017",5,550,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",6,60,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",7,700,initial) | ("Tue Jan 31 00:00:00 2017",7,770,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",8,800,initial) | ("Tue Feb 28 00:00:00 2017",8,880,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",9,90,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
(9 rows)
SELECT * FROM pa_target ORDER BY tid;
diff --git a/src/test/regress/expected/returning.out b/src/test/regress/expected/returning.out
new file mode 100644
index cb51bb8..b4888db
--- a/src/test/regress/expected/returning.out
+++ b/src/test/regress/expected/returning.out
@@ -355,3 +355,511 @@ INSERT INTO foo AS bar DEFAULT VALUES RE
42
(1 row)
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+ERROR: syntax error at or near "nonsuch"
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS so...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+ERROR: table name "foo" specified more than once
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+ERROR: OLD cannot be specified multiple times
+LINE 1: ...EFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) ...
+ ^
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ -> Result
+ Output: 4, NULL::text, 42, '99'::bigint
+(4 rows)
+
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+------+----+----+----+----+----------+-------+----+----+----+----+----+----+----+----
+ | | | | | | foo | (0,4) | 4 | | 42 | 99 | 4 | | 42 | 99
+(1 row)
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (o.tableoid)::regclass, o.ctid, o.f1, o.f2, o.f3, o.f4, (n.tableoid)::regclass, n.ctid, n.f1, n.f2, n.f3, n.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ Conflict Resolution: UPDATE
+ Conflict Arbiter Indexes: foo_f1_idx
+ -> Values Scan on "*VALUES*"
+ Output: "*VALUES*".column1, "*VALUES*".column2, 42, '99'::bigint
+(6 rows)
+
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+----+----------+-------+----+------------+----+----+----+------------+----+----
+ foo | (0,4) | 4 | | 42 | 99 | foo | (0,5) | 4 | conflicted | -1 | 99 | 4 | conflicted | -1 | 99
+ | | | | | | foo | (0,6) | 5 | ok | 42 | 99 | 5 | ok | 42 | 99
+(2 rows)
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ QUERY PLAN
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, old.*, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, new.*, (((old.f4)::text || '->'::text) || (new.f4)::text)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(8 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ tableoid | ctid | f1 | f2 | f3 | f4 | old | tableoid | ctid | f1 | f2 | f3 | f4 | new | change
+----------+-------+----+----+----+----+--------------+----------+-------+----+----+----+-----+---------------+---------
+ foo | (0,6) | 5 | ok | 42 | 99 | (5,ok,42,99) | foo | (0,7) | 5 | ok | 42 | 100 | (5,ok,42,100) | 99->100
+(1 row)
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(6 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+-----+----------+------+----+----+----+----+----+----+----+-----
+ foo | (0,7) | 5 | ok | 42 | 100 | | | | | | | 5 | ok | 42 | 100
+(1 row)
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ -> Result
+ Output: 5, 'subquery test'::text, 42, '99'::bigint
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(16 rows)
+
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ | 109
+(1 row)
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2), (SubPlan 3)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Result
+ Output: (old.f4 = new.f4)
+ SubPlan 2
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 3
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(23 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ ?column? | old_max | new_max
+----------+---------+---------
+ f | 109 | 110
+(1 row)
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(18 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ 110 |
+(1 row)
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, new.f1, new.f2, new.f3, new.f4, foo_2.f1, foo_2.f2, foo_2.f3, foo_2.f4
+ Update on pg_temp.foo foo_2
+ -> Nested Loop
+ Output: (foo_2.f2 || ' (deleted)'::text), '-1'::integer, '-1'::bigint, foo_1.ctid, foo_1.tableoid, foo_2.tableoid, foo_2.ctid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f2, foo_2.f1, foo_2.tableoid, foo_2.ctid
+ Filter: (foo_2.f1 = 4)
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.ctid, foo_1.f1, foo_1.tableoid
+ Filter: (foo_1.f1 = 4)
+(11 rows)
+
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----+------------+----+----+----+----------------------+----+----+----+----------------------+----+----
+ 4 | conflicted | -1 | 99 | 4 | conflicted (deleted) | -1 | -1 | 4 | conflicted (deleted) | -1 | -1
+(1 row)
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, old.(joinme.other), new.f1, new.f2, new.f3, new.f4, new.(joinme.other), foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4, joinme.other, (new.f3 - old.f3)
+ Update on pg_temp.foo foo_1
+ -> Hash Join
+ Output: foo_2.f1, (foo_2.f3 + 1), joinme.ctid, foo_2.ctid, joinme_1.ctid, joinme.other, foo_1.tableoid, foo_1.ctid, foo_2.tableoid
+ Hash Cond: (foo_1.f2 = joinme.f2j)
+ -> Hash Join
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme_1.ctid, joinme_1.f2j
+ Hash Cond: (joinme_1.f2j = foo_1.f2)
+ -> Seq Scan on pg_temp.joinme joinme_1
+ Output: joinme_1.ctid, joinme_1.f2j
+ -> Hash
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Hash
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Hash Join
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Hash Cond: (joinme.f2j = foo_2.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.ctid, joinme.other, joinme.f2j
+ -> Hash
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Filter: (foo_2.f3 = 57)
+(27 rows)
+
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 57 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 1
+(1 row)
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ERROR: column reference "new.f1" is ambiguous
+LINE 3: RETURNING new.f1, new.f4
+ ^
+DETAIL: It could refer to either a PL/pgSQL variable or a table column.
+QUERY: UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4
+CONTEXT: PL/pgSQL function joinview_upd_trig_fn() line 4 at SQL statement
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.joinview
+ Output: old.f1, old.f2, old.f3, old.f4, old.other, new.f1, new.f2, new.f3, new.f4, new.other, joinview.f1, joinview.f2, joinview.f3, joinview.f4, joinview.other, (new.f3 - old.f3)
+ -> Hash Join
+ Output: (foo.f3 + 1), '7'::bigint, ROW(foo.f1, foo.f2, foo.f3, foo.f4, joinme.other), foo.ctid, joinme.ctid, foo.tableoid
+ Hash Cond: (joinme.f2j = foo.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.other, joinme.ctid, joinme.f2j
+ -> Hash
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ -> Seq Scan on pg_temp.foo
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ Filter: (foo.f3 = 58)
+(12 rows)
+
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 1
+(1 row)
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+NOTICE: drop cascades to 3 other objects
+DETAIL: drop cascades to rule voo_i on view voo
+drop cascades to view joinview
+drop cascades to rule foo_del_rule on table foo
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+ERROR: column old.f3 does not exist
+LINE 1: UPDATE foo SET f4 = f4 + 1 RETURNING old.f3;
+ ^
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+ old | new
+-------------------------------+------------------------------
+ (1,xxx,20) | (1,xxx,21)
+ (2,more,141) | (2,more,142)
+ (4,"conflicted (deleted)",-1) | (4,"conflicted (deleted)",0)
+ (3,zoo2,70) | (3,zoo2,71)
+(4 rows)
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+------+----------+-------+-------
+ | | zerocol | (0,1) | (0,1)
+(1 row)
+
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+-------+----------+------+-------
+ zerocol | (0,1) | | | (0,1)
+(1 row)
+
+DROP TABLE zerocol;
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+----------+------+---+---+---+-------------+-------+---+------+----+---+------+----
+ | | | | | foo_part_s1 | (0,1) | 1 | 17.1 | P1 | 1 | 17.1 | P1
+ | | | | | foo_part_s2 | (0,1) | 2 | 17.2 | P2 | 2 | 17.2 | P2
+ | | | | | foo_part_d1 | (0,1) | 3 | 17.3 | P3 | 3 | 17.3 | P3
+ | | | | | foo_part_d2 | (0,1) | 4 | 17.4 | P4 | 4 | 17.4 | P4
+(4 rows)
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_s1 | (0,1) | 1 | 17.1 | P1 | foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | 2 | 18.1 | P1->P2
+(1 row)
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_d1 | (0,1) | 3 | 17.3 | P3 | foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | 1 | 18.3 | P3->P1
+(1 row)
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+--------+-------------+-------+---+------+------------+---+------+------------
+ foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | 3 | 19.3 | P3->P1->P3
+(1 row)
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+------------+-------------+-------+---+------+----------------+---+------+----------------
+ foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | 4 | 20.3 | P3->P1->P3->P4
+(1 row)
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----------------+----------+------+---+---+---+---+------+----------------
+ foo_part_s2 | (0,1) | 2 | 17.2 | P2 | | | | | | 2 | 17.2 | P2
+ foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | | | | | | 2 | 18.1 | P1->P2
+ foo_part_d2 | (0,1) | 4 | 17.4 | P4 | | | | | | 4 | 17.4 | P4
+ foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | | | | | | 4 | 20.3 | P3->P1->P3->P4
+(4 rows)
+
+DROP TABLE foo_parted;
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o)
+ o.*, new.*, o, new, o.f1 = new.f1, o = new,
+ (SELECT o.f2 = new.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = new.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = new);
+END;
+\sf foo_update
+CREATE OR REPLACE FUNCTION public.foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ UPDATE foo SET f1 = (foo.f1 + 1)
+ RETURNING WITH (OLD AS o) o.f1,
+ o.f2,
+ o.f4,
+ new.f1,
+ new.f2,
+ new.f4,
+ o.*::foo AS o,
+ new.*::foo AS new,
+ (o.f1 = new.f1),
+ (o.* = new.*),
+ ( SELECT (o.f2 = new.f2)),
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f1 = o.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f4 = new.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = o.*)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = new.*)) AS count;
+END
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
new file mode 100644
index 2b47013..c4ebd67
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3645,7 +3645,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1
-- test deparsing
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3684,11 +3687,12 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
CREATE OR REPLACE FUNCTION public.merge_sf_test()
- RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[])
+ RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[], old_id integer, old_data text, old_filling integer[], new_id integer, new_data text, new_filling integer[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3726,12 +3730,18 @@ BEGIN ATOMIC
WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
- RETURNING MERGE_ACTION() AS action,
+ RETURNING WITH (OLD AS o, NEW AS n) MERGE_ACTION() AS action,
s.a,
s.b,
t.id,
t.data,
- t.filling;
+ t.filling,
+ o.id,
+ o.data,
+ o.filling,
+ n.id,
+ n.data,
+ n.filling;
END
CREATE FUNCTION merge_sf_test2()
RETURNS void
diff --git a/src/test/regress/expected/updatable_views.out b/src/test/regress/expected/updatable_views.out
new file mode 100644
index 8786058..bface0e
--- a/src/test/regress/expected/updatable_views.out
+++ b/src/test/regress/expected/updatable_views.out
@@ -437,7 +437,7 @@ NOTICE: drop cascades to view ro_view19
-- simple updatable view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS SELECT *, 'Const' AS c FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view1';
@@ -462,7 +462,8 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | YES
rw_view1 | b | YES
-(2 rows)
+ rw_view1 | c | NO
+(3 rows)
INSERT INTO rw_view1 VALUES (3, 'Row 3');
INSERT INTO rw_view1 (a) VALUES (4);
@@ -479,20 +480,22 @@ SELECT * FROM base_tbl;
5 | Unspecified
(6 rows)
+SET jit_above_cost = 0;
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+-------+---+-------------
- UPDATE | 1 | ROW 1 | 1 | ROW 1
- DELETE | 3 | ROW 3 | 3 | Row 3
- INSERT | 2 | ROW 2 | 2 | Unspecified
+ RETURNING merge_action(), v.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c
+--------------+---+-------+-------------------+-----------------------+---+-------------+-------
+ UPDATE | 1 | ROW 1 | (1,"Row 1",Const) | (1,"ROW 1",Const) | 1 | ROW 1 | Const
+ DELETE | 3 | ROW 3 | (3,"Row 3",Const) | | 3 | Row 3 | Const
+ INSERT | 2 | ROW 2 | | (2,Unspecified,Const) | 2 | Unspecified | Const
(3 rows)
+SET jit_above_cost TO DEFAULT;
SELECT * FROM base_tbl ORDER BY a;
a | b
----+-------------
@@ -511,13 +514,13 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------------
- UPDATE | 1 | R1 | 1 | R1
- DELETE | | | 5 | Unspecified
- DELETE | 2 | R2 | 2 | Unspecified
- INSERT | 3 | R3 | 3 | Unspecified
+ RETURNING merge_action(), v.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c
+--------------+---+----+-----------------------+-----------------------+---+-------------+-------
+ UPDATE | 1 | R1 | (1,"ROW 1",Const) | (1,R1,Const) | 1 | R1 | Const
+ DELETE | | | (5,Unspecified,Const) | | 5 | Unspecified | Const
+ DELETE | 2 | R2 | (2,Unspecified,Const) | | 2 | Unspecified | Const
+ INSERT | 3 | R3 | | (3,Unspecified,Const) | 3 | Unspecified | Const
(4 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -634,8 +637,10 @@ DROP TABLE base_tbl_hist;
-- view on top of view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view2';
@@ -660,27 +665,29 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view2 | aaa | YES
rw_view2 | bbb | YES
-(2 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(4 rows)
INSERT INTO rw_view2 VALUES (3, 'Row 3');
INSERT INTO rw_view2 (aaa) VALUES (4);
SELECT * FROM rw_view2;
- aaa | bbb
------+-------------
- 1 | Row 1
- 2 | Row 2
- 3 | Row 3
- 4 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Unspecified | Const1 | Const2
(4 rows)
UPDATE rw_view2 SET bbb='Row 4' WHERE aaa=4;
DELETE FROM rw_view2 WHERE aaa=2;
SELECT * FROM rw_view2;
- aaa | bbb
------+-------
- 1 | Row 1
- 3 | Row 3
- 4 | Row 4
+ aaa | bbb | c1 | c2
+-----+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Row 4 | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -688,20 +695,20 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-------------
- DELETE | 3 | R3 | 3 | Row 3
- UPDATE | 4 | R4 | 4 | R4
- INSERT | 5 | R5 | 5 | Unspecified
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+---------------------------+-------------------------------+-----+-------------+--------+--------
+ DELETE | 3 | R3 | (3,"Row 3",Const1,Const2) | | 3 | Row 3 | Const1 | Const2
+ UPDATE | 4 | R4 | (4,"Row 4",Const1,Const2) | (4,R4,Const1,Const2) | 4 | R4 | Const1 | Const2
+ INSERT | 5 | R5 | | (5,Unspecified,Const1,Const2) | 5 | Unspecified | Const1 | Const2
(3 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-------------
- 1 | Row 1
- 4 | R4
- 5 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 4 | R4 | Const1 | Const2
+ 5 | Unspecified | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -710,21 +717,21 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-----------------------
- UPDATE | | | 1 | Not matched by source
- DELETE | 4 | r4 | 4 | R4
- UPDATE | 5 | r5 | 5 | r5
- INSERT | 6 | r6 | 6 | Unspecified
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+-------------------------------+-------------------------------------------+-----+-----------------------+--------+--------
+ UPDATE | | | (1,"Row 1",Const1,Const2) | (1,"Not matched by source",Const1,Const2) | 1 | Not matched by source | Const1 | Const2
+ DELETE | 4 | r4 | (4,R4,Const1,Const2) | | 4 | R4 | Const1 | Const2
+ UPDATE | 5 | r5 | (5,Unspecified,Const1,Const2) | (5,r5,Const1,Const2) | 5 | r5 | Const1 | Const2
+ INSERT | 6 | r6 | | (6,Unspecified,Const1,Const2) | 6 | Unspecified | Const1 | Const2
(4 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-----------------------
- 1 | Not matched by source
- 5 | r5
- 6 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-----------------------+--------+--------
+ 1 | Not matched by source | Const1 | Const2
+ 5 | r5 | Const1 | Const2
+ 6 | Unspecified | Const1 | Const2
(3 rows)
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -886,16 +893,25 @@ SELECT table_name, column_name, is_updat
rw_view2 | b | YES
(4 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | a | b
+---+---+---+-------
+ | | 3 | Row 3
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+ a | b | a | b
+---+----+---+----
+ 3 | R3 | 3 | R3
+(1 row)
+
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+----+---+-----------
+ 3 | R3 | 3 | Row three
(1 row)
SELECT * FROM rw_view2;
@@ -906,10 +922,10 @@ SELECT * FROM rw_view2;
3 | Row three
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+-----------+---+---
+ 3 | Row three | |
(1 row)
SELECT * FROM rw_view2;
@@ -960,8 +976,10 @@ drop cascades to view rw_view2
-- view on top of view with triggers
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name LIKE 'rw_view%'
@@ -992,9 +1010,12 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE FUNCTION rw_view1_trig_fn()
RETURNS trigger AS
@@ -1002,9 +1023,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -1045,9 +1068,12 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_upd_trig INSTEAD OF UPDATE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1081,9 +1107,12 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_del_trig INSTEAD OF DELETE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1117,41 +1146,44 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+---+----+----+---+-------+----------------+--------
+ | | | | 3 | Row 3 | Trigger Const1 | Const2
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-------+--------+--------+---+-----------+----------------+--------
+ 3 | Row 3 | Const1 | Const2 | 3 | Row three | Trigger Const1 | Const2
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-----------
- 1 | Row 1
- 2 | Row 2
- 3 | Row three
+ a | b | c1 | c2
+---+-----------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row three | Const1 | Const2
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-----------+--------+--------+---+---+----+----
+ 3 | Row three | Const1 | Const2 | | | |
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-------
- 1 | Row 1
- 2 | Row 2
+ a | b | c1 | c2
+---+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
(2 rows)
MERGE INTO rw_view2 t
@@ -1159,12 +1191,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------
- DELETE | 1 | R1 | 1 | Row 1
- UPDATE | 2 | R2 | 2 | R2
- INSERT | 3 | R3 | 3 | R3
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+---------------------------+--------------------------------+---+-------+----------------+--------
+ DELETE | 1 | R1 | (1,"Row 1",Const1,Const2) | | 1 | Row 1 | Const1 | Const2
+ UPDATE | 2 | R2 | (2,"Row 2",Const1,Const2) | (2,R2,"Trigger Const1",Const2) | 2 | R2 | Trigger Const1 | Const2
+ INSERT | 3 | R3 | | (3,R3,"Trigger Const1",Const2) | 3 | R3 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -1182,12 +1214,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-----------------------
- UPDATE | 2 | r2 | 2 | r2
- UPDATE | | | 3 | Not matched by source
- INSERT | 1 | r1 | 1 | r1
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+----------------------+-----------------------------------------------------+---+-----------------------+----------------+--------
+ UPDATE | 2 | r2 | (2,R2,Const1,Const2) | (2,r2,"Trigger Const1",Const2) | 2 | r2 | Trigger Const1 | Const2
+ UPDATE | | | (3,R3,Const1,Const2) | (3,"Not matched by source","Trigger Const1",Const2) | 3 | Not matched by source | Trigger Const1 | Const2
+ INSERT | 1 | r1 | | (1,r1,"Trigger Const1",Const2) | 1 | r1 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql
new file mode 100644
index ce9981d..9d98053
--- a/src/test/regress/sql/merge.sql
+++ b/src/test/regress/sql/merge.sql
@@ -235,7 +235,7 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -677,7 +677,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 T
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -930,7 +930,9 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -956,7 +958,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -970,7 +972,7 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -988,7 +990,7 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
ROLLBACK;
@@ -1265,7 +1267,7 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
@@ -1456,7 +1458,7 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
diff --git a/src/test/regress/sql/returning.sql b/src/test/regress/sql/returning.sql
new file mode 100644
index a460f82..29841a9
--- a/src/test/regress/sql/returning.sql
+++ b/src/test/regress/sql/returning.sql
@@ -160,3 +160,205 @@ INSERT INTO foo AS bar DEFAULT VALUES RE
INSERT INTO foo AS bar DEFAULT VALUES RETURNING foo.*; -- fails, wrong name
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.*; -- ok
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.f3; -- ok
+
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DROP TABLE zerocol;
+
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DROP TABLE foo_parted;
+
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o)
+ o.*, new.*, o, new, o.f1 = new.f1, o = new,
+ (SELECT o.f2 = new.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = new.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = new);
+END;
+
+\sf foo_update
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
new file mode 100644
index 4a5fa50..fdd3ff1
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1294,7 +1294,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -1333,7 +1336,8 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
diff --git a/src/test/regress/sql/updatable_views.sql b/src/test/regress/sql/updatable_views.sql
new file mode 100644
index 93b693a..1f8b0ff
--- a/src/test/regress/sql/updatable_views.sql
+++ b/src/test/regress/sql/updatable_views.sql
@@ -154,7 +154,7 @@ DROP SEQUENCE uv_seq CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS SELECT *, 'Const' AS c FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -175,13 +175,18 @@ UPDATE rw_view1 SET a=5 WHERE a=4;
DELETE FROM rw_view1 WHERE b='Row 2';
SELECT * FROM base_tbl;
+SET jit_above_cost = 0;
+
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, t.*;
+
+SET jit_above_cost TO DEFAULT;
+
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view1 t
@@ -191,7 +196,7 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view1 SET a=6 WHERE a=5;
@@ -240,8 +245,10 @@ DROP TABLE base_tbl_hist;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -268,7 +275,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
MERGE INTO rw_view2 t
@@ -277,7 +284,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -362,10 +369,14 @@ SELECT table_name, column_name, is_updat
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t USING (VALUES (3, 'Row 3')) AS v(a,b) ON t.a = v.a
@@ -381,8 +392,10 @@ DROP TABLE base_tbl CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -407,9 +420,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -479,10 +494,10 @@ SELECT table_name, column_name, is_updat
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t
@@ -490,7 +505,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view2 t
@@ -498,7 +513,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
new file mode 100644
index a65e1c0..c82b6db
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2464,6 +2464,10 @@ RestrictInfo
Result
ResultRelInfo
ResultState
+ReturningClause
+ReturningExpr
+ReturningOption
+ReturningOptionKind
ReturnSetInfo
ReturnStmt
RevmapContents
@@ -2612,6 +2616,7 @@ SetOperationStmt
SetQuantifier
SetToDefault
SetupWorkerPtrType
+SetVarReturningType_context
ShDependObjectInfo
SharedAggInfo
SharedBitmapHeapInstrumentation
@@ -3082,6 +3087,7 @@ VariableSpace
VariableStatData
VariableSubstituteHook
Variables
+VarReturningType
Vector32
Vector8
VersionedQuery
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2024-10-14 15:34 ` jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: jian he @ 2024-10-14 15:34 UTC (permalink / raw)
To: Dean Rasheed <[email protected]>; +Cc: Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Mon, Oct 14, 2024 at 7:03 PM Dean Rasheed <[email protected]> wrote:
>
> > > typedef struct ReturningOption
> > > {
> > > NodeTag type;
> > > bool isNew;
> > > char *name;
> > > int location;
> > > } ReturningOption;
>
> Thinking about that struct some more, I think "isNew" is better done
> as an enum, since this is meant to be a generic option. So even though
> it might never have more than 2 possible values, I think it's neater
> done that way.
>
typedef struct ReturningOption
{
NodeTag type;
ReturningOptionKind option; /* specified option */
char *value; /* option's value */
ParseLoc location; /* token location, or -1 if unknown */
} ReturningOption;
@@ -4304,6 +4332,16 @@ raw_expression_tree_walker_impl(Node *no
return true;
}
break;
+ case T_ReturningClause:
+ {
+ ReturningClause *returning = (ReturningClause *) node;
+
+ if (WALK(returning->options))
+ return true;
+ if (WALK(returning->exprs))
+ return true;
+ }
+ break;
+ if (WALK(returning->options))
+ return true;
T_ReturningOption is primitive, so we only need to
"if (WALK(returning->exprs))"?
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
@ 2024-10-14 16:05 ` Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2024-10-14 16:05 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Mon, 14 Oct 2024 at 16:34, jian he <[email protected]> wrote:
>
> typedef struct ReturningOption
> {
> NodeTag type;
> ReturningOptionKind option; /* specified option */
> char *value; /* option's value */
> ParseLoc location; /* token location, or -1 if unknown */
> } ReturningOption;
>
> @@ -4304,6 +4332,16 @@ raw_expression_tree_walker_impl(Node *no
> return true;
> }
> break;
> + case T_ReturningClause:
> + {
> + ReturningClause *returning = (ReturningClause *) node;
> +
> + if (WALK(returning->options))
> + return true;
> + if (WALK(returning->exprs))
> + return true;
> + }
> + break;
>
> + if (WALK(returning->options))
> + return true;
> T_ReturningOption is primitive, so we only need to
> "if (WALK(returning->exprs))"?
No, it still needs to walk the options so that it will call the
callback for each option. The fact that T_ReturningOption is primitive
doesn't change that, it just means that there is no more structure
*below* a ReturningOption that needs to be traversed. The
ReturningOption itself still needs to be traversed. For example,
imagine you wanted to use raw_expression_tree_walker() to print out
the whole structure of a raw parse tree. You'd want your printing
callback to be called for every node, including the ReturningOption
nodes.
Regards,
Dean
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2024-10-29 13:05 ` Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2024-10-29 13:05 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
Rebased version attached. No other changes.
Regards,
Dean
Attachments:
[text/x-patch] support-returning-old-new-v21.patch (240.7K, ../../CAEZATCUh2+GH5fnF=2noNb14F4W7KfucxsDR3=x1kJ_ax2F-wg@mail.gmail.com/2-support-returning-old-new-v21.patch)
download | inline diff:
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
new file mode 100644
index f2bcd6a..701e6b5
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -4975,12 +4975,12 @@ INSERT INTO ft2 (c1,c2,c3) SELECT c1+100
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
- c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
-------+-----+-----+----+----+----+------------+----
- 1101 | 201 | aaa | | | | ft2 |
- 1102 | 202 | bbb | | | | ft2 |
- 1103 | 203 | ccc | | | | ft2 |
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
+ old | new | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+-----+---------------------------------+----+----+----+----+----+----+----+----+------+-----+-----+----+----+----+------------+----
+ | (1101,201,aaa,,,,"ft2 ",) | | | | | | | | | 1101 | 201 | aaa | | | | ft2 |
+ | (1102,202,bbb,,,,"ft2 ",) | | | | | | | | | 1102 | 202 | bbb | | | | ft2 |
+ | (1103,203,ccc,,,,"ft2 ",) | | | | | | | | | 1103 | 203 | ccc | | | | ft2 |
(3 rows)
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
@@ -5111,6 +5111,31 @@ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 ||
1017 | 507 | 0001700017_update7 | | | | ft2 |
(102 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.c1, old.c2, old.c3, old.c4, old.c5, old.c6, old.c7, old.c8, new.c1, new.c2, new.c3, new.c4, new.c5, new.c6, new.c7, new.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c2 = $2, c3 = $3 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: (c2 + 400), (c3 || '_update7b'::text), ctid, ft2.*
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 7)) FOR UPDATE
+(6 rows)
+
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+----+-----+---------------+------------------------------+--------------------------+----+------------+-----+----+-----+------------------------+------------------------------+--------------------------+----+------------+-----
+ 7 | 407 | 00007_update7 | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo | 7 | 807 | 00007_update7_update7b | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo
+ 17 | 407 | 00017_update7 | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo | 17 | 807 | 00017_update7_update7b | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo
+ 27 | 407 | 00027_update7 | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo | 27 | 807 | 00027_update7_update7b | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo
+ 37 | 407 | 00037_update7 | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo | 37 | 807 | 00037_update7_update7b | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -5241,6 +5266,29 @@ DELETE FROM ft2 WHERE c1 % 10 = 5 RETURN
1105 |
(103 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------
+ Delete on public.ft2
+ Output: old.c1, c4
+ Remote SQL: DELETE FROM "S 1"."T 1" WHERE ctid = $1 RETURNING "C 1", c4
+ -> Foreign Scan on public.ft2
+ Output: ctid
+ Remote SQL: SELECT ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 6)) FOR UPDATE
+(6 rows)
+
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ c1 | c4
+----+------------------------------
+ 6 | Wed Jan 07 00:00:00 1970 PST
+ 16 | Sat Jan 17 00:00:00 1970 PST
+ 26 | Tue Jan 27 00:00:00 1970 PST
+ 36 | Fri Feb 06 00:00:00 1970 PST
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
QUERY PLAN
@@ -6165,6 +6213,70 @@ UPDATE ft2 SET c3 = 'foo'
(1296,96,foo,,,,"ft2 ",) | 1296 | 96 | foo | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
(16 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.*, new.*, ft2.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan
+ Output: 'bar'::text, ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Relations: ((public.ft2) INNER JOIN (public.ft4)) INNER JOIN (public.ft5)
+ Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2.c1, r2.c2, r2.c3) END, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r2.c1, r2.c2, r2.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r2 ON (((r1.c2 = r2.c1)) AND ((r1."C 1" > 1200)))) INNER JOIN "S 1"."T 4" r3 ON (((r2.c1 = r3.c1)))) FOR UPDATE OF r1
+ -> Nested Loop
+ Output: ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Join Filter: (ft4.c1 = ft5.c1)
+ -> Sort
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Sort Key: ft2.c2
+ -> Hash Join
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Hash Cond: (ft2.c2 = ft4.c1)
+ -> Foreign Scan on public.ft2
+ Output: ft2.ctid, ft2.*, ft2.c2
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" > 1200)) FOR UPDATE
+ -> Hash
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ -> Foreign Scan on public.ft4
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 3"
+ -> Materialize
+ Output: ft5.*, ft5.c1
+ -> Foreign Scan on public.ft5
+ Output: ft5.*, ft5.c1
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 4"
+(29 rows)
+
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ old | new | ft2 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | ft4 | c1 | c2 | c3
+--------------------------------+--------------------------------+--------------------------------+------+----+-----+----+----+----+------------+----+----------------+----+----+--------
+ (1206,6,foo,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | 1206 | 6 | bar | | | | ft2 | | (6,7,AAA006) | 6 | 7 | AAA006
+ (1212,12,foo,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | 1212 | 12 | bar | | | | ft2 | | (12,13,AAA012) | 12 | 13 | AAA012
+ (1224,24,foo,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | 1224 | 24 | bar | | | | ft2 | | (24,25,AAA024) | 24 | 25 | AAA024
+ (1230,30,foo,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | 1230 | 30 | bar | | | | ft2 | | (30,31,AAA030) | 30 | 31 | AAA030
+ (1242,42,foo,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | 1242 | 42 | bar | | | | ft2 | | (42,43,AAA042) | 42 | 43 | AAA042
+ (1248,48,foo,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | 1248 | 48 | bar | | | | ft2 | | (48,49,AAA048) | 48 | 49 | AAA048
+ (1260,60,foo,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | 1260 | 60 | bar | | | | ft2 | | (60,61,AAA060) | 60 | 61 | AAA060
+ (1266,66,foo,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | 1266 | 66 | bar | | | | ft2 | | (66,67,AAA066) | 66 | 67 | AAA066
+ (1278,78,foo,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | 1278 | 78 | bar | | | | ft2 | | (78,79,AAA078) | 78 | 79 | AAA078
+ (1284,84,foo,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | 1284 | 84 | bar | | | | ft2 | | (84,85,AAA084) | 84 | 85 | AAA084
+ (1296,96,foo,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | 1296 | 96 | bar | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
+ (1218,18,foo,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | 1218 | 18 | bar | | | | ft2 | | (18,19,AAA018) | 18 | 19 | AAA018
+ (1236,36,foo,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | 1236 | 36 | bar | | | | ft2 | | (36,37,AAA036) | 36 | 37 | AAA036
+ (1254,54,foo,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | 1254 | 54 | bar | | | | ft2 | | (54,55,AAA054) | 54 | 55 | AAA054
+ (1272,72,foo,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | 1272 | 72 | bar | | | | ft2 | | (72,73,AAA072) | 72 | 73 | AAA072
+ (1290,90,foo,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | 1290 | 90 | bar | | | | ft2 | | (90,91,AAA090) | 90 | 91 | AAA090
+(16 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
new file mode 100644
index 372fe6d..c704dae
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1469,7 +1469,7 @@ EXPLAIN (verbose, costs off)
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 300, c3 = c3 || '_update3' WHERE c1 % 10 = 3; -- can be pushed down
@@ -1477,6 +1477,13 @@ UPDATE ft2 SET c2 = c2 + 300, c3 = c3 ||
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *; -- can be pushed down
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -1485,6 +1492,11 @@ UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = f
EXPLAIN (verbose, costs off)
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4; -- can be pushed down
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2;
@@ -1511,6 +1523,17 @@ UPDATE ft2 SET c3 = 'foo'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
RETURNING ft2, ft2.*, ft4, ft4.*;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/doc/src/sgml/dml.sgml b/doc/src/sgml/dml.sgml
new file mode 100644
index 3d95bdb..458aee7
--- a/doc/src/sgml/dml.sgml
+++ b/doc/src/sgml/dml.sgml
@@ -308,7 +308,8 @@ DELETE FROM products;
</para>
<para>
- In an <command>INSERT</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>INSERT</command>, the default data available to
+ <literal>RETURNING</literal> is
the row as it was inserted. This is not so useful in trivial inserts,
since it would just repeat the data provided by the client. But it can
be very handy when relying on computed default values. For example,
@@ -325,7 +326,8 @@ INSERT INTO users (firstname, lastname)
</para>
<para>
- In an <command>UPDATE</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>UPDATE</command>, the default data available to
+ <literal>RETURNING</literal> is
the new content of the modified row. For example:
<programlisting>
UPDATE products SET price = price * 1.10
@@ -335,7 +337,8 @@ UPDATE products SET price = price * 1.10
</para>
<para>
- In a <command>DELETE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>DELETE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the deleted row. For example:
<programlisting>
DELETE FROM products
@@ -345,7 +348,8 @@ DELETE FROM products
</para>
<para>
- In a <command>MERGE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>MERGE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the source row plus the content of the inserted, updated, or
deleted target row. Since it is quite common for the source and target to
have many of the same columns, specifying <literal>RETURNING *</literal>
@@ -360,6 +364,35 @@ MERGE INTO products p USING new_products
</para>
<para>
+ In each of these commands, it is also possible to explicitly return the
+ old and new content of the modified row. For example:
+<programlisting>
+UPDATE products SET price = price * 1.10
+ WHERE price <= 99.99
+ RETURNING name, old.price AS old_price, new.price AS new_price,
+ new.price - old.price AS price_change;
+</programlisting>
+ In this example, writing <literal>new.price</literal> is the same as
+ just writing <literal>price</literal>, but it makes the meaning clearer.
+ </para>
+
+ <para>
+ This syntax for returning old and new values is available in
+ <command>INSERT</command>, <command>UPDATE</command>,
+ <command>DELETE</command>, and <command>MERGE</command> commands, but
+ typically old values will be <literal>NULL</literal> for an
+ <command>INSERT</command>, and new values will be <literal>NULL</literal>
+ for a <command>DELETE</command>. However, there are situations where it
+ can still be useful for those commands. For example, in an
+ <command>INSERT</command> with an
+ <link linkend="sql-on-conflict"><literal>ON CONFLICT DO UPDATE</literal></link>
+ clause, the old values will be non-<literal>NULL</literal> for conflicting
+ rows. Similarly, if a <command>DELETE</command> is turned into an
+ <command>UPDATE</command> by a <link linkend="sql-createrule">rewrite rule</link>,
+ the new values may be non-<literal>NULL</literal>.
+ </para>
+
+ <para>
If there are triggers (<xref linkend="triggers"/>) on the target table,
the data available to <literal>RETURNING</literal> is the row as modified by
the triggers. Thus, inspecting columns computed by triggers is another
diff --git a/doc/src/sgml/ref/delete.sgml b/doc/src/sgml/ref/delete.sgml
new file mode 100644
index 7717855..29649f6
--- a/doc/src/sgml/ref/delete.sgml
+++ b/doc/src/sgml/ref/delete.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
[ USING <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -161,6 +162,26 @@ DELETE FROM [ ONLY ] <replaceable class=
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -170,6 +191,23 @@ DELETE FROM [ ONLY ] <replaceable class=
or table(s) listed in <literal>USING</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return old values.
+ </para>
+
+ <para>
+ For a simple <command>DELETE</command>, all new values will be
+ <literal>NULL</literal>. However, if an <literal>ON DELETE</literal>
+ rule causes an <command>INSERT</command> or <command>UPDATE</command>
+ to be executed instead, the new values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml
new file mode 100644
index 6f0adee..3f13991
--- a/doc/src/sgml/ref/insert.sgml
+++ b/doc/src/sgml/ref/insert.sgml
@@ -26,7 +26,8 @@ INSERT INTO <replaceable class="paramete
[ OVERRIDING { SYSTEM | USER } VALUE ]
{ DEFAULT VALUES | VALUES ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="parameter">query</replaceable> }
[ ON CONFLICT [ <replaceable class="parameter">conflict_target</replaceable> ] <replaceable class="parameter">conflict_action</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">conflict_target</replaceable> can be one of:</phrase>
@@ -294,6 +295,26 @@ INSERT INTO <replaceable class="paramete
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -305,6 +326,23 @@ INSERT INTO <replaceable class="paramete
<literal>*</literal> to return all columns of the inserted or updated
row(s).
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values.
+ </para>
+
+ <para>
+ For a simple <command>INSERT</command>, all old values will be
+ <literal>NULL</literal>. However, for an <command>INSERT</command>
+ with an <literal>ON CONFLICT DO UPDATE</literal> clause, the old
+ values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
@@ -714,6 +752,20 @@ INSERT INTO distributors (did, dname)
</programlisting>
</para>
<para>
+ Insert or update new distributors as above, returning information
+ about any existing values that were updated, together with the new data
+ inserted. Note that the returned values for <literal>old_did</literal>
+ and <literal>old_dname</literal> will be <literal>NULL</literal> for
+ non-conflicting rows:
+<programlisting>
+INSERT INTO distributors (did, dname)
+ VALUES (5, 'Gizmo Transglobal'), (6, 'Associated Computing, Inc')
+ ON CONFLICT (did) DO UPDATE SET dname = EXCLUDED.dname
+ RETURNING old.did AS old_did, old.dname AS old_dname,
+ new.did AS new_did, new.dname AS new_dname;
+</programlisting>
+ </para>
+ <para>
Insert a distributor, or do nothing for rows proposed for insertion
when an existing, excluded row (a row with a matching constrained
column or columns after before row insert triggers fire) exists.
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
new file mode 100644
index 97b34b9..1b47e9a
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
MERGE INTO [ ONLY ] <replaceable class="parameter">target_table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">target_alias</replaceable> ]
USING <replaceable class="parameter">data_source</replaceable> ON <replaceable class="parameter">join_condition</replaceable>
<replaceable class="parameter">when_clause</replaceable> [...]
-[ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+[ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">data_source</replaceable> is:</phrase>
@@ -500,6 +501,25 @@ DELETE
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -517,6 +537,17 @@ DELETE
qualifying the <literal>*</literal> with the name or alias of the source
or target table.
</para>
+ <para>
+ A column name or <literal>*</literal> may also be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values from the target table to be returned. An unqualified column
+ name, or <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values
+ for <literal>INSERT</literal> and <literal>UPDATE</literal> actions, and
+ old values for <literal>DELETE</literal> actions.
+ </para>
</listitem>
</varlistentry>
@@ -739,7 +770,7 @@ WHEN MATCHED AND w.stock + s.stock_delta
UPDATE SET stock = w.stock + s.stock_delta
WHEN MATCHED THEN
DELETE
-RETURNING merge_action(), w.*;
+RETURNING merge_action(), w.winename, old.stock AS old_stock, new.stock AS new_stock;
</programlisting>
The <literal>wine_stock_changes</literal> table might be, for example, a
diff --git a/doc/src/sgml/ref/update.sgml b/doc/src/sgml/ref/update.sgml
new file mode 100644
index 1c433be..12ec5ba
--- a/doc/src/sgml/ref/update.sgml
+++ b/doc/src/sgml/ref/update.sgml
@@ -29,7 +29,8 @@ UPDATE [ ONLY ] <replaceable class="para
} [, ...]
[ FROM <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -212,6 +213,26 @@ UPDATE [ ONLY ] <replaceable class="para
</varlistentry>
<varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
<para>
@@ -221,6 +242,16 @@ UPDATE [ ONLY ] <replaceable class="para
or table(s) listed in <literal>FROM</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return new values.
+ </para>
</listitem>
</varlistentry>
@@ -348,12 +379,13 @@ UPDATE weather SET temp_lo = temp_lo+1,
</para>
<para>
- Perform the same operation and return the updated entries:
+ Perform the same operation and return the updated entries, and the old
+ precipitation value:
<programlisting>
UPDATE weather SET temp_lo = temp_lo+1, temp_hi = temp_lo+15, prcp = DEFAULT
WHERE city = 'San Francisco' AND date = '2003-07-03'
- RETURNING temp_lo, temp_hi, prcp;
+ RETURNING temp_lo, temp_hi, prcp, old.prcp AS old_prcp;
</programlisting>
</para>
diff --git a/doc/src/sgml/rules.sgml b/doc/src/sgml/rules.sgml
new file mode 100644
index 7a928bd..e992baa
--- a/doc/src/sgml/rules.sgml
+++ b/doc/src/sgml/rules.sgml
@@ -1646,6 +1646,23 @@ CREATE RULE shoelace_ins AS ON INSERT TO
</para>
<para>
+ Note that in the <literal>RETURNING</literal> clause of a rule,
+ <literal>OLD</literal> and <literal>NEW</literal> refer to the
+ pseudorelations added as extra range table entries to the rewritten
+ query, rather than old/new rows in the result relation. Thus, for
+ example, in a rule supporting <command>UPDATE</command> queries on this
+ view, if the <literal>RETURNING</literal> clause contained
+ <literal>old.sl_name</literal>, the old name would always be returned,
+ regardless of whether the <literal>RETURNING</literal> clause in the
+ query on the view specified <literal>OLD</literal> or <literal>NEW</literal>,
+ which might be confusing. To avoid this confusion, and support returning
+ old and new values in queries on the view, the <literal>RETURNING</literal>
+ clause in the rule definition should refer to entries from the result
+ relation such as <literal>shoelace_data.sl_name</literal>, without
+ specifying <literal>OLD</literal> or <literal>NEW</literal>.
+ </para>
+
+ <para>
Now assume that once in a while, a pack of shoelaces arrives at
the shop and a big parts list along with it. But you don't want
to manually update the <literal>shoelace</literal> view every
diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c
new file mode 100644
index 45954d9..697f895
--- a/src/backend/executor/execExpr.c
+++ b/src/backend/executor/execExpr.c
@@ -55,10 +55,15 @@
typedef struct ExprSetupInfo
{
- /* Highest attribute numbers fetched from inner/outer/scan tuple slots: */
+ /*
+ * Highest attribute numbers fetched from inner/outer/scan/old/new tuple
+ * slots:
+ */
AttrNumber last_inner;
AttrNumber last_outer;
AttrNumber last_scan;
+ AttrNumber last_old;
+ AttrNumber last_new;
/* MULTIEXPR SubPlan nodes appearing in the expression: */
List *multiexpr_subplans;
} ExprSetupInfo;
@@ -446,8 +451,25 @@ ExecBuildProjectionInfo(List *targetList
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+
+ /*
+ * Get the tuple from the relation being scanned, and keep
+ * track of whether any OLD/NEW values were requested.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_ASSIGN_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_ASSIGN_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ default:
+ scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+ break;
+ }
break;
}
@@ -535,7 +557,7 @@ ExecBuildUpdateProjection(List *targetLi
int nAssignableCols;
bool sawJunk;
Bitmapset *assignedCols;
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
ExprEvalStep scratch = {0};
int outerattnum;
ListCell *lc,
@@ -924,6 +946,7 @@ ExecInitExprRec(Expr *node, ExprState *s
/* system column */
scratch.d.var.attnum = variable->varattno;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -936,7 +959,20 @@ ExecInitExprRec(Expr *node, ExprState *s
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_SYSVAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ default:
+ scratch.opcode = EEOP_SCAN_SYSVAR;
+ break;
+ }
break;
}
}
@@ -945,6 +981,7 @@ ExecInitExprRec(Expr *node, ExprState *s
/* regular user column */
scratch.d.var.attnum = variable->varattno - 1;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -957,7 +994,20 @@ ExecInitExprRec(Expr *node, ExprState *s
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_VAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ default:
+ scratch.opcode = EEOP_SCAN_VAR;
+ break;
+ }
break;
}
}
@@ -2567,6 +2617,28 @@ ExecInitExprRec(Expr *node, ExprState *s
break;
}
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ int retstep;
+
+ /* Skip expression evaluation if OLD/NEW row doesn't exist */
+ scratch.opcode = EEOP_RETURNINGEXPR;
+ scratch.d.returningexpr.nullflag = rexpr->retold ?
+ EEO_FLAG_OLD_IS_NULL : EEO_FLAG_NEW_IS_NULL;
+ scratch.d.returningexpr.jumpdone = -1; /* set below */
+ ExprEvalPushStep(state, &scratch);
+ retstep = state->steps_len - 1;
+
+ /* Steps to evaluate expression to return */
+ ExecInitExprRec(rexpr->retexpr, state, resv, resnull);
+
+ /* Jump target used if OLD/NEW row doesn't exist */
+ state->steps[retstep].d.returningexpr.jumpdone = state->steps_len;
+
+ break;
+ }
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(node));
@@ -2778,7 +2850,7 @@ ExecInitSubPlanExpr(SubPlan *subplan,
static void
ExecCreateExprSetupSteps(ExprState *state, Node *node)
{
- ExprSetupInfo info = {0, 0, 0, NIL};
+ ExprSetupInfo info = {0, 0, 0, 0, 0, NIL};
/* Prescan to find out what we need. */
expr_setup_walker(node, &info);
@@ -2801,8 +2873,8 @@ ExecPushExprSetupSteps(ExprState *state,
scratch.resnull = NULL;
/*
- * Add steps deforming the ExprState's inner/outer/scan slots as much as
- * required by any Vars appearing in the expression.
+ * Add steps deforming the ExprState's inner/outer/scan/old/new slots as
+ * much as required by any Vars appearing in the expression.
*/
if (info->last_inner > 0)
{
@@ -2834,6 +2906,26 @@ ExecPushExprSetupSteps(ExprState *state,
if (ExecComputeSlotInfo(state, &scratch))
ExprEvalPushStep(state, &scratch);
}
+ if (info->last_old > 0)
+ {
+ scratch.opcode = EEOP_OLD_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_old;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
+ if (info->last_new > 0)
+ {
+ scratch.opcode = EEOP_NEW_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_new;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
/*
* Add steps to execute any MULTIEXPR SubPlans appearing in the
@@ -2880,7 +2972,18 @@ expr_setup_walker(Node *node, ExprSetupI
/* INDEX_VAR is handled by default case */
default:
- info->last_scan = Max(info->last_scan, attnum);
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ info->last_old = Max(info->last_old, attnum);
+ break;
+ case VAR_RETURNING_NEW:
+ info->last_new = Max(info->last_new, attnum);
+ break;
+ default:
+ info->last_scan = Max(info->last_scan, attnum);
+ break;
+ }
break;
}
return false;
@@ -2919,6 +3022,11 @@ expr_setup_walker(Node *node, ExprSetupI
* evaluation of the expression will have the same type of slot, with an
* equivalent descriptor.
*
+ * EEOP_OLD_FETCHSOME and EEOP_NEW_FETCHSOME are used to process RETURNING, if
+ * OLD/NEW columns are referred to explicitly. In both cases, the tuple
+ * descriptor comes from the parent scan node, so we treat them the same as
+ * EEOP_SCAN_FETCHSOME.
+ *
* Returns true if the deforming step is required, false otherwise.
*/
static bool
@@ -2932,7 +3040,9 @@ ExecComputeSlotInfo(ExprState *state, Ex
Assert(opcode == EEOP_INNER_FETCHSOME ||
opcode == EEOP_OUTER_FETCHSOME ||
- opcode == EEOP_SCAN_FETCHSOME);
+ opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME);
if (op->d.fetch.known_desc != NULL)
{
@@ -2984,7 +3094,9 @@ ExecComputeSlotInfo(ExprState *state, Ex
desc = ExecGetResultType(os);
}
}
- else if (opcode == EEOP_SCAN_FETCHSOME)
+ else if (opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME)
{
desc = parent->scandesc;
@@ -3032,6 +3144,12 @@ ExecInitWholeRowVar(ExprEvalStep *scratc
scratch->d.wholerow.tupdesc = NULL; /* filled at runtime */
scratch->d.wholerow.junkFilter = NULL;
+ /* update ExprState flags if Var refers to OLD/NEW */
+ if (variable->varreturningtype == VAR_RETURNING_OLD)
+ state->flags |= EEO_FLAG_HAS_OLD;
+ else if (variable->varreturningtype == VAR_RETURNING_NEW)
+ state->flags |= EEO_FLAG_HAS_NEW;
+
/*
* If the input tuple came from a subquery, it might contain "resjunk"
* columns (such as GROUP BY or ORDER BY columns), which we don't want to
@@ -3534,7 +3652,7 @@ ExecBuildAggTrans(AggState *aggstate, Ag
PlanState *parent = &aggstate->ss.ps;
ExprEvalStep scratch = {0};
bool isCombine = DO_AGGSPLIT_COMBINE(aggstate->aggsplit);
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
state->expr = (Expr *) aggstate;
state->parent = parent;
@@ -4240,6 +4358,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = latt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4248,6 +4367,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = ratt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4374,6 +4494,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4382,6 +4503,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
new file mode 100644
index 6a7f18f..c66ac61
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -157,17 +157,25 @@ static void ExecEvalRowNullInt(ExprState
static Datum ExecJustInnerVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustOuterVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustScanVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustOldVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustNewVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignInnerVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignOuterVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignScanVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignOldVar(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignNewVar(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustApplyFuncToCase(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustConst(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustInnerVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustOuterVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustScanVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignInnerVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignOuterVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
static Datum ExecJustAssignScanVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
+static Datum ExecJustAssignNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull);
/* execution helper functions */
static pg_attribute_always_inline void ExecAggPlainTransByVal(AggState *aggstate,
@@ -296,6 +304,18 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustScanVar;
return;
}
+ else if (step0 == EEOP_OLD_FETCHSOME &&
+ step1 == EEOP_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustOldVar;
+ return;
+ }
+ else if (step0 == EEOP_NEW_FETCHSOME &&
+ step1 == EEOP_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustNewVar;
+ return;
+ }
else if (step0 == EEOP_INNER_FETCHSOME &&
step1 == EEOP_ASSIGN_INNER_VAR)
{
@@ -314,6 +334,18 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustAssignScanVar;
return;
}
+ else if (step0 == EEOP_OLD_FETCHSOME &&
+ step1 == EEOP_ASSIGN_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignOldVar;
+ return;
+ }
+ else if (step0 == EEOP_NEW_FETCHSOME &&
+ step1 == EEOP_ASSIGN_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignNewVar;
+ return;
+ }
else if (step0 == EEOP_CASE_TESTVAL &&
step1 == EEOP_FUNCEXPR_STRICT &&
state->steps[0].d.casetest.value)
@@ -346,6 +378,16 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustScanVarVirt;
return;
}
+ else if (step0 == EEOP_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustOldVarVirt;
+ return;
+ }
+ else if (step0 == EEOP_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustNewVarVirt;
+ return;
+ }
else if (step0 == EEOP_ASSIGN_INNER_VAR)
{
state->evalfunc_private = (void *) ExecJustAssignInnerVarVirt;
@@ -361,6 +403,16 @@ ExecReadyInterpretedExpr(ExprState *stat
state->evalfunc_private = (void *) ExecJustAssignScanVarVirt;
return;
}
+ else if (step0 == EEOP_ASSIGN_OLD_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignOldVarVirt;
+ return;
+ }
+ else if (step0 == EEOP_ASSIGN_NEW_VAR)
+ {
+ state->evalfunc_private = (void *) ExecJustAssignNewVarVirt;
+ return;
+ }
}
#if defined(EEO_USE_COMPUTED_GOTO)
@@ -400,6 +452,8 @@ ExecInterpExpr(ExprState *state, ExprCon
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
/*
* This array has to be in the same order as enum ExprEvalOp.
@@ -410,16 +464,24 @@ ExecInterpExpr(ExprState *state, ExprCon
&&CASE_EEOP_INNER_FETCHSOME,
&&CASE_EEOP_OUTER_FETCHSOME,
&&CASE_EEOP_SCAN_FETCHSOME,
+ &&CASE_EEOP_OLD_FETCHSOME,
+ &&CASE_EEOP_NEW_FETCHSOME,
&&CASE_EEOP_INNER_VAR,
&&CASE_EEOP_OUTER_VAR,
&&CASE_EEOP_SCAN_VAR,
+ &&CASE_EEOP_OLD_VAR,
+ &&CASE_EEOP_NEW_VAR,
&&CASE_EEOP_INNER_SYSVAR,
&&CASE_EEOP_OUTER_SYSVAR,
&&CASE_EEOP_SCAN_SYSVAR,
+ &&CASE_EEOP_OLD_SYSVAR,
+ &&CASE_EEOP_NEW_SYSVAR,
&&CASE_EEOP_WHOLEROW,
&&CASE_EEOP_ASSIGN_INNER_VAR,
&&CASE_EEOP_ASSIGN_OUTER_VAR,
&&CASE_EEOP_ASSIGN_SCAN_VAR,
+ &&CASE_EEOP_ASSIGN_OLD_VAR,
+ &&CASE_EEOP_ASSIGN_NEW_VAR,
&&CASE_EEOP_ASSIGN_TMP,
&&CASE_EEOP_ASSIGN_TMP_MAKE_RO,
&&CASE_EEOP_CONST,
@@ -461,6 +523,7 @@ ExecInterpExpr(ExprState *state, ExprCon
&&CASE_EEOP_SQLVALUEFUNCTION,
&&CASE_EEOP_CURRENTOFEXPR,
&&CASE_EEOP_NEXTVALUEEXPR,
+ &&CASE_EEOP_RETURNINGEXPR,
&&CASE_EEOP_ARRAYEXPR,
&&CASE_EEOP_ARRAYCOERCE,
&&CASE_EEOP_ROW,
@@ -529,6 +592,8 @@ ExecInterpExpr(ExprState *state, ExprCon
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
#if defined(EEO_USE_COMPUTED_GOTO)
EEO_DISPATCH();
@@ -568,6 +633,24 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, oldslot);
+
+ slot_getsomeattrs(oldslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, newslot);
+
+ slot_getsomeattrs(newslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_VAR)
{
int attnum = op->d.var.attnum;
@@ -611,6 +694,32 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ *op->resvalue = oldslot->tts_values[attnum];
+ *op->resnull = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ *op->resvalue = newslot->tts_values[attnum];
+ *op->resnull = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_SYSVAR)
{
ExecEvalSysVar(state, op, econtext, innerslot);
@@ -629,6 +738,18 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, oldslot);
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, newslot);
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_WHOLEROW)
{
/* too complex for an inline implementation */
@@ -688,6 +809,40 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_ASSIGN_OLD_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = oldslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_ASSIGN_NEW_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = newslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ASSIGN_TMP)
{
int resultnum = op->d.assign_tmp.resultnum;
@@ -1364,6 +1519,23 @@ ExecInterpExpr(ExprState *state, ExprCon
EEO_NEXT();
}
+ EEO_CASE(EEOP_RETURNINGEXPR)
+ {
+ /*
+ * The next op actually evaluates the expression. If the OLD/NEW
+ * row doesn't exist, skip that and return NULL.
+ */
+ if (state->flags & op->d.returningexpr.nullflag)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+
+ EEO_JUMP(op->d.returningexpr.jumpdone);
+ }
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ARRAYEXPR)
{
/* too complex for an inline implementation */
@@ -2045,10 +2217,14 @@ CheckExprStillValid(ExprState *state, Ex
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
for (int i = 0; i < state->steps_len; i++)
{
@@ -2079,6 +2255,22 @@ CheckExprStillValid(ExprState *state, Ex
CheckVarSlotCompatibility(scanslot, attnum + 1, op->d.var.vartype);
break;
}
+
+ case EEOP_OLD_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(oldslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
+
+ case EEOP_NEW_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(newslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
default:
break;
}
@@ -2253,7 +2445,7 @@ get_cached_rowtype(Oid type_id, int32 ty
* Fast-path functions, for very simple expressions
*/
-/* implementation of ExecJust(Inner|Outer|Scan)Var */
+/* implementation of ExecJust(Inner|Outer|Scan|Old|New)Var */
static pg_attribute_always_inline Datum
ExecJustVarImpl(ExprState *state, TupleTableSlot *slot, bool *isnull)
{
@@ -2291,7 +2483,21 @@ ExecJustScanVar(ExprState *state, ExprCo
return ExecJustVarImpl(state, econtext->ecxt_scantuple, isnull);
}
-/* implementation of ExecJustAssign(Inner|Outer|Scan)Var */
+/* Simple reference to OLD Var in RETURNING */
+static Datum
+ExecJustOldVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Simple reference to NEW Var in RETURNING */
+static Datum
+ExecJustNewVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
+/* implementation of ExecJustAssign(Inner|Outer|Scan|Old|New)Var */
static pg_attribute_always_inline Datum
ExecJustAssignVarImpl(ExprState *state, TupleTableSlot *inslot, bool *isnull)
{
@@ -2338,6 +2544,20 @@ ExecJustAssignScanVar(ExprState *state,
return ExecJustAssignVarImpl(state, econtext->ecxt_scantuple, isnull);
}
+/* Evaluate OLD Var and assign to appropriate column of result tuple */
+static Datum
+ExecJustAssignOldVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Evaluate NEW Var and assign to appropriate column of result tuple */
+static Datum
+ExecJustAssignNewVar(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
/* Evaluate CASE_TESTVAL and apply a strict function to it */
static Datum
ExecJustApplyFuncToCase(ExprState *state, ExprContext *econtext, bool *isnull)
@@ -2386,7 +2606,7 @@ ExecJustConst(ExprState *state, ExprCont
return op->d.constval.value;
}
-/* implementation of ExecJust(Inner|Outer|Scan)VarVirt */
+/* implementation of ExecJust(Inner|Outer|Scan|Old|New)VarVirt */
static pg_attribute_always_inline Datum
ExecJustVarVirtImpl(ExprState *state, TupleTableSlot *slot, bool *isnull)
{
@@ -2429,7 +2649,21 @@ ExecJustScanVarVirt(ExprState *state, Ex
return ExecJustVarVirtImpl(state, econtext->ecxt_scantuple, isnull);
}
-/* implementation of ExecJustAssign(Inner|Outer|Scan)VarVirt */
+/* Like ExecJustOldVar, optimized for virtual slots */
+static Datum
+ExecJustOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarVirtImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Like ExecJustNewVar, optimized for virtual slots */
+static Datum
+ExecJustNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustVarVirtImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
+/* implementation of ExecJustAssign(Inner|Outer|Scan|Old|New)VarVirt */
static pg_attribute_always_inline Datum
ExecJustAssignVarVirtImpl(ExprState *state, TupleTableSlot *inslot, bool *isnull)
{
@@ -2472,6 +2706,20 @@ ExecJustAssignScanVarVirt(ExprState *sta
return ExecJustAssignVarVirtImpl(state, econtext->ecxt_scantuple, isnull);
}
+/* Like ExecJustAssignOldVar, optimized for virtual slots */
+static Datum
+ExecJustAssignOldVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarVirtImpl(state, econtext->ecxt_oldtuple, isnull);
+}
+
+/* Like ExecJustAssignNewVar, optimized for virtual slots */
+static Datum
+ExecJustAssignNewVarVirt(ExprState *state, ExprContext *econtext, bool *isnull)
+{
+ return ExecJustAssignVarVirtImpl(state, econtext->ecxt_newtuple, isnull);
+}
+
#if defined(EEO_USE_COMPUTED_GOTO)
/*
* Comparator used when building address->opcode lookup table for
@@ -4922,8 +5170,40 @@ ExecEvalWholeRowVar(ExprState *state, Ex
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- slot = econtext->ecxt_scantuple;
+
+ /*
+ * Get the tuple from the relation being scanned.
+ *
+ * By default, this uses the "scan" tuple slot, but a wholerow Var
+ * in the RETURNING list may explicitly refer to OLD/NEW. If the
+ * OLD/NEW row doesn't exist, we just return NULL.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ Assert(state->flags & EEO_FLAG_HAS_OLD);
+ if (state->flags & EEO_FLAG_OLD_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_oldtuple;
+ break;
+ case VAR_RETURNING_NEW:
+ Assert(state->flags & EEO_FLAG_HAS_NEW);
+ if (state->flags & EEO_FLAG_NEW_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_newtuple;
+ break;
+ default:
+ slot = econtext->ecxt_scantuple;
+ break;
+ }
break;
}
@@ -5126,6 +5406,38 @@ ExecEvalSysVar(ExprState *state, ExprEva
{
Datum d;
+ /*
+ * For OLD/NEW system attributes, check whether the OLD/NEW row exists. If
+ * it doesn't, the OLD/NEW system attribute is NULL.
+ */
+ if (op->d.var.varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ bool rowIsNull;
+
+ switch (op->d.var.varreturningtype)
+ {
+ case VAR_RETURNING_OLD:
+ Assert(state->flags & EEO_FLAG_HAS_OLD);
+ rowIsNull = (state->flags & EEO_FLAG_OLD_IS_NULL) != 0;
+ break;
+ case VAR_RETURNING_NEW:
+ Assert(state->flags & EEO_FLAG_HAS_NEW);
+ rowIsNull = (state->flags & EEO_FLAG_NEW_IS_NULL) != 0;
+ break;
+ default:
+ elog(ERROR, "unrecognized varreturningtype: %d",
+ (int) op->d.var.varreturningtype);
+ rowIsNull = false; /* keep compiler quiet */
+ }
+
+ if (rowIsNull)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ }
+
/* slot_getsysattr has sufficient defenses against bad attnums */
d = slot_getsysattr(slot,
op->d.var.attnum,
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
new file mode 100644
index cc9a594..594fc97
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -1255,6 +1255,7 @@ InitResultRelInfo(ResultRelInfo *resultR
resultRelInfo->ri_ReturningSlot = NULL;
resultRelInfo->ri_TrigOldSlot = NULL;
resultRelInfo->ri_TrigNewSlot = NULL;
+ resultRelInfo->ri_AllNullSlot = NULL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_MATCHED] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_SOURCE] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_TARGET] = NIL;
diff --git a/src/backend/executor/execUtils.c b/src/backend/executor/execUtils.c
new file mode 100644
index 740e8fb..a47882c
--- a/src/backend/executor/execUtils.c
+++ b/src/backend/executor/execUtils.c
@@ -1200,6 +1200,34 @@ ExecGetReturningSlot(EState *estate, Res
}
/*
+ * Return a relInfo's all-NULL tuple slot for processing returning tuples.
+ *
+ * Note: this slot is intentionally filled with NULLs in every column, and
+ * should be considered read-only --- the caller must not update it.
+ */
+TupleTableSlot *
+ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo)
+{
+ if (relInfo->ri_AllNullSlot == NULL)
+ {
+ Relation rel = relInfo->ri_RelationDesc;
+ MemoryContext oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
+ TupleTableSlot *slot;
+
+ slot = ExecInitExtraTupleSlot(estate,
+ RelationGetDescr(rel),
+ table_slot_callbacks(rel));
+ ExecStoreAllNullTuple(slot);
+
+ relInfo->ri_AllNullSlot = slot;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ return relInfo->ri_AllNullSlot;
+}
+
+/*
* Return the map needed to convert given child result relation's tuples to
* the rowtype of the query's main target ("root") relation. Note that a
* NULL result is valid and means that no conversion is needed.
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
new file mode 100644
index 1161520..e84e47e
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -102,6 +102,13 @@ typedef struct ModifyTableContext
TM_FailureData tmfd;
/*
+ * The tuple deleted when doing a cross-partition UPDATE with a RETURNING
+ * clause that refers to OLD columns (converted to the root's tuple
+ * descriptor).
+ */
+ TupleTableSlot *cpDeletedSlot;
+
+ /*
* The tuple projected by the INSERT's RETURNING clause, when doing a
* cross-partition UPDATE
*/
@@ -243,34 +250,67 @@ ExecCheckPlanOutput(Relation resultRel,
/*
* ExecProcessReturning --- evaluate a RETURNING list
*
+ * context: context for the ModifyTable operation
* resultRelInfo: current result rel
- * tupleSlot: slot holding tuple actually inserted/updated/deleted
+ * cmdType: operation/merge action performed (INSERT, UPDATE, or DELETE)
+ * oldSlot: slot holding old tuple deleted or updated
+ * newSlot: slot holding new tuple inserted or updated
* planSlot: slot holding tuple returned by top subplan node
*
- * Note: If tupleSlot is NULL, the FDW should have already provided econtext's
- * scan tuple.
+ * Note: If oldSlot and newSlot are NULL, the FDW should have already provided
+ * econtext's scan tuple and its old & new tuples are not needed (FDW direct-
+ * modify is disabled if the RETURNING list refers to any OLD/NEW values).
*
* Returns a slot holding the result tuple
*/
static TupleTableSlot *
-ExecProcessReturning(ResultRelInfo *resultRelInfo,
- TupleTableSlot *tupleSlot,
+ExecProcessReturning(ModifyTableContext *context,
+ ResultRelInfo *resultRelInfo,
+ CmdType cmdType,
+ TupleTableSlot *oldSlot,
+ TupleTableSlot *newSlot,
TupleTableSlot *planSlot)
{
+ EState *estate = context->estate;
ProjectionInfo *projectReturning = resultRelInfo->ri_projectReturning;
ExprContext *econtext = projectReturning->pi_exprContext;
/* Make tuple and any needed join variables available to ExecProject */
- if (tupleSlot)
- econtext->ecxt_scantuple = tupleSlot;
+ if (cmdType == CMD_DELETE && oldSlot)
+ econtext->ecxt_scantuple = oldSlot;
+ if (cmdType != CMD_DELETE && newSlot)
+ econtext->ecxt_scantuple = newSlot;
econtext->ecxt_outertuple = planSlot;
+ /* Make old/new tuples available to ExecProject, if required */
+ if (oldSlot)
+ econtext->ecxt_oldtuple = oldSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ econtext->ecxt_oldtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_oldtuple = NULL; /* No references to OLD columns */
+
+ if (newSlot)
+ econtext->ecxt_newtuple = newSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW)
+ econtext->ecxt_newtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_newtuple = NULL; /* No references to NEW columns */
+
/*
- * RETURNING expressions might reference the tableoid column, so
- * reinitialize tts_tableOid before evaluating them.
+ * Tell ExecProject whether or not the OLD/NEW rows actually exist. This
+ * information is required to evaluate ReturningExpr nodes and also in
+ * ExecEvalSysVar and ExecEvalWholeRowVar.
*/
- econtext->ecxt_scantuple->tts_tableOid =
- RelationGetRelid(resultRelInfo->ri_RelationDesc);
+ if (oldSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_OLD_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_OLD_IS_NULL;
+
+ if (newSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_NEW_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_NEW_IS_NULL;
/* Compute the RETURNING expressions */
return ExecProject(projectReturning);
@@ -1204,7 +1244,56 @@ ExecInsert(ModifyTableContext *context,
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- result = ExecProcessReturning(resultRelInfo, slot, planSlot);
+ {
+ TupleTableSlot *oldSlot = NULL;
+
+ /*
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD columns, ExecDelete() will have saved the tuple
+ * deleted from the original partition, which we must use here to
+ * compute the OLD column values. Otherwise, all OLD column values
+ * will be NULL.
+ */
+ if (context->cpDeletedSlot)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the OLD tuple to the new partition's format/slot, if
+ * needed. Note that ExceDelete() already converted it to the
+ * root's partition's format/slot.
+ */
+ oldSlot = context->cpDeletedSlot;
+ tupconv_map = ExecGetRootToChildMap(resultRelInfo, estate);
+ if (tupconv_map != NULL)
+ {
+ oldSlot = execute_attr_map_slot(tupconv_map->attrMap,
+ oldSlot,
+ ExecGetReturningSlot(estate,
+ resultRelInfo));
+
+ oldSlot->tts_tableOid = context->cpDeletedSlot->tts_tableOid;
+ ItemPointerCopy(&context->cpDeletedSlot->tts_tid, &oldSlot->tts_tid);
+ }
+ }
+
+ result = ExecProcessReturning(context, resultRelInfo, CMD_INSERT,
+ oldSlot, slot, planSlot);
+
+ /*
+ * For a cross-partition UPDATE, release the old tuple, first making
+ * sure that the result slot has a local copy of any pass-by-reference
+ * values.
+ */
+ if (context->cpDeletedSlot)
+ {
+ ExecMaterializeSlot(result);
+ ExecClearTuple(oldSlot);
+ if (context->cpDeletedSlot != oldSlot)
+ ExecClearTuple(context->cpDeletedSlot);
+ context->cpDeletedSlot = NULL;
+ }
+ }
if (inserted_tuple)
*inserted_tuple = slot;
@@ -1442,6 +1531,7 @@ ExecDelete(ModifyTableContext *context,
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
TupleTableSlot *slot = NULL;
TM_Result result;
+ bool saveOld;
if (tupleDeleted)
*tupleDeleted = false;
@@ -1676,8 +1766,17 @@ ldelete:
ExecDeleteEpilogue(context, resultRelInfo, tupleid, oldtuple, changingPart);
- /* Process RETURNING if present and if requested */
- if (processReturning && resultRelInfo->ri_projectReturning)
+ /*
+ * Process RETURNING if present and if requested.
+ *
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD column values, save the old tuple here for later
+ * processing of the RETURNING list by ExecInsert().
+ */
+ saveOld = changingPart && resultRelInfo->ri_projectReturning &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD;
+
+ if (resultRelInfo->ri_projectReturning && (processReturning || saveOld))
{
/*
* We have to put the target tuple into a slot, which means first we
@@ -1705,7 +1804,41 @@ ldelete:
}
}
- rslot = ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ /*
+ * If required, save the old tuple for later processing of the
+ * RETURNING list by ExecInsert().
+ */
+ if (saveOld)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the tuple into the root partition's format/slot, if
+ * needed. ExecInsert() will then convert it to the new
+ * partition's format/slot, if necessary.
+ */
+ tupconv_map = ExecGetChildToRootMap(resultRelInfo);
+ if (tupconv_map != NULL)
+ {
+ ResultRelInfo *rootRelInfo = context->mtstate->rootResultRelInfo;
+ TupleTableSlot *oldSlot = slot;
+
+ slot = execute_attr_map_slot(tupconv_map->attrMap,
+ slot,
+ ExecGetReturningSlot(estate,
+ rootRelInfo));
+
+ slot->tts_tableOid = oldSlot->tts_tableOid;
+ ItemPointerCopy(&oldSlot->tts_tid, &slot->tts_tid);
+ }
+
+ context->cpDeletedSlot = slot;
+
+ return NULL;
+ }
+
+ rslot = ExecProcessReturning(context, resultRelInfo, CMD_DELETE,
+ slot, NULL, context->planSlot);
/*
* Before releasing the target tuple again, make sure rslot has a
@@ -1758,6 +1891,7 @@ ExecCrossPartitionUpdate(ModifyTableCont
bool tuple_deleted;
TupleTableSlot *epqslot = NULL;
+ context->cpDeletedSlot = NULL;
context->cpUpdateReturningSlot = NULL;
*retry_slot = NULL;
@@ -2258,6 +2392,7 @@ ExecCrossPartitionUpdateForeignKey(Modif
* the planSlot. oldtuple is passed to foreign table triggers; it is
* NULL when the foreign table has no relevant triggers.
*
+ * oldSlot contains the old tuple value.
* slot contains the new tuple value to be stored.
* planSlot is the output of the ModifyTable's subplan; we use it
* to access values from other input tables (for RETURNING),
@@ -2270,8 +2405,8 @@ ExecCrossPartitionUpdateForeignKey(Modif
*/
static TupleTableSlot *
ExecUpdate(ModifyTableContext *context, ResultRelInfo *resultRelInfo,
- ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot,
- bool canSetTag)
+ ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *oldSlot,
+ TupleTableSlot *slot, bool canSetTag)
{
EState *estate = context->estate;
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
@@ -2389,7 +2524,6 @@ redo_act:
{
TupleTableSlot *inputslot;
TupleTableSlot *epqslot;
- TupleTableSlot *oldSlot;
if (IsolationUsesXactSnapshot())
ereport(ERROR,
@@ -2504,7 +2638,8 @@ redo_act:
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- return ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ return ExecProcessReturning(context, resultRelInfo, CMD_UPDATE,
+ oldSlot, slot, context->planSlot);
return NULL;
}
@@ -2724,16 +2859,23 @@ ExecOnConflictUpdate(ModifyTableContext
/* Execute UPDATE with projection */
*returning = ExecUpdate(context, resultRelInfo,
- conflictTid, NULL,
+ conflictTid, NULL, existing,
resultRelInfo->ri_onConflict->oc_ProjSlot,
canSetTag);
/*
* Clear out existing tuple, as there might not be another conflict among
* the next input rows. Don't want to hold resources till the end of the
- * query.
+ * query. First though, make sure that the returning slot, if any, has a
+ * local copy of any OLD pass-by-reference values, if it refers to any OLD
+ * columns.
*/
+ if (*returning != NULL &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ ExecMaterializeSlot(*returning);
+
ExecClearTuple(existing);
+
return true;
}
@@ -3338,13 +3480,20 @@ lmerge_matched:
switch (commandType)
{
case CMD_UPDATE:
- rslot = ExecProcessReturning(resultRelInfo, newslot,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_UPDATE,
+ resultRelInfo->ri_oldTupleSlot,
+ newslot,
context->planSlot);
break;
case CMD_DELETE:
- rslot = ExecProcessReturning(resultRelInfo,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_DELETE,
resultRelInfo->ri_oldTupleSlot,
+ NULL,
context->planSlot);
break;
@@ -3894,6 +4043,7 @@ ExecModifyTable(PlanState *pstate)
if (node->mt_merge_pending_not_matched != NULL)
{
context.planSlot = node->mt_merge_pending_not_matched;
+ context.cpDeletedSlot = NULL;
slot = ExecMergeNotMatched(&context, node->resultRelInfo,
node->canSetTag);
@@ -3913,6 +4063,7 @@ ExecModifyTable(PlanState *pstate)
/* Fetch the next row from subplan */
context.planSlot = ExecProcNode(subplanstate);
+ context.cpDeletedSlot = NULL;
/* No more tuples to process? */
if (TupIsNull(context.planSlot))
@@ -3980,9 +4131,15 @@ ExecModifyTable(PlanState *pstate)
* A scan slot containing the data that was actually inserted,
* updated or deleted has already been made available to
* ExecProcessReturning by IterateDirectModify, so no need to
- * provide it here.
+ * provide it here. The individual old and new slots are not
+ * needed, since direct-modify is disabled if the RETURNING list
+ * refers to OLD/NEW values.
*/
- slot = ExecProcessReturning(resultRelInfo, NULL, context.planSlot);
+ Assert((resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD) == 0 &&
+ (resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW) == 0);
+
+ slot = ExecProcessReturning(&context, resultRelInfo, operation,
+ NULL, NULL, context.planSlot);
return slot;
}
@@ -4172,7 +4329,7 @@ ExecModifyTable(PlanState *pstate)
/* Now apply the update. */
slot = ExecUpdate(&context, resultRelInfo, tupleid, oldtuple,
- slot, node->canSetTag);
+ oldSlot, slot, node->canSetTag);
if (tuplock)
UnlockTuple(resultRelInfo->ri_RelationDesc, tupleid,
InplaceUpdateTupleLock);
diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c
new file mode 100644
index 0b3b574..18f9ad4
--- a/src/backend/jit/llvm/llvmjit_expr.c
+++ b/src/backend/jit/llvm/llvmjit_expr.c
@@ -105,6 +105,8 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_innerslot;
LLVMValueRef v_outerslot;
LLVMValueRef v_scanslot;
+ LLVMValueRef v_oldslot;
+ LLVMValueRef v_newslot;
LLVMValueRef v_resultslot;
/* nulls/values of slots */
@@ -114,6 +116,10 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_outernulls;
LLVMValueRef v_scanvalues;
LLVMValueRef v_scannulls;
+ LLVMValueRef v_oldvalues;
+ LLVMValueRef v_oldnulls;
+ LLVMValueRef v_newvalues;
+ LLVMValueRef v_newnulls;
LLVMValueRef v_resultvalues;
LLVMValueRef v_resultnulls;
@@ -200,6 +206,16 @@ llvm_compile_expr(ExprState *state)
v_econtext,
FIELDNO_EXPRCONTEXT_OUTERTUPLE,
"v_outerslot");
+ v_oldslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_OLDTUPLE,
+ "v_oldslot");
+ v_newslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_NEWTUPLE,
+ "v_newslot");
v_resultslot = l_load_struct_gep(b,
StructExprState,
v_state,
@@ -237,6 +253,26 @@ llvm_compile_expr(ExprState *state)
v_outerslot,
FIELDNO_TUPLETABLESLOT_ISNULL,
"v_outernulls");
+ v_oldvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_oldvalues");
+ v_oldnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_oldnulls");
+ v_newvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_newvalues");
+ v_newnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_newnulls");
v_resultvalues = l_load_struct_gep(b,
StructTupleTableSlot,
v_resultslot,
@@ -302,6 +338,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_FETCHSOME:
case EEOP_OUTER_FETCHSOME:
case EEOP_SCAN_FETCHSOME:
+ case EEOP_OLD_FETCHSOME:
+ case EEOP_NEW_FETCHSOME:
{
TupleDesc desc = NULL;
LLVMValueRef v_slot;
@@ -326,8 +364,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_FETCHSOME)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_FETCHSOME)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_FETCHSOME)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
/*
* Check if all required attributes are available, or
@@ -396,6 +438,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_VAR:
case EEOP_OUTER_VAR:
case EEOP_SCAN_VAR:
+ case EEOP_OLD_VAR:
+ case EEOP_NEW_VAR:
{
LLVMValueRef value,
isnull;
@@ -413,11 +457,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
v_attnum = l_int32_const(lc, op->d.var.attnum);
value = l_load_gep1(b, TypeSizeT, v_values, v_attnum, "");
@@ -432,6 +486,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_SYSVAR:
case EEOP_OUTER_SYSVAR:
case EEOP_SCAN_SYSVAR:
+ case EEOP_OLD_SYSVAR:
+ case EEOP_NEW_SYSVAR:
{
LLVMValueRef v_slot;
@@ -439,8 +495,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_SYSVAR)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_SYSVAR)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_SYSVAR)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
build_EvalXFunc(b, mod, "ExecEvalSysVar",
v_state, op, v_econtext, v_slot);
@@ -458,6 +518,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_ASSIGN_INNER_VAR:
case EEOP_ASSIGN_OUTER_VAR:
case EEOP_ASSIGN_SCAN_VAR:
+ case EEOP_ASSIGN_OLD_VAR:
+ case EEOP_ASSIGN_NEW_VAR:
{
LLVMValueRef v_value;
LLVMValueRef v_isnull;
@@ -478,11 +540,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_ASSIGN_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_ASSIGN_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
/* load data */
v_attnum = l_int32_const(lc, op->d.assign_var.attnum);
@@ -1639,6 +1711,45 @@ llvm_compile_expr(ExprState *state)
LLVMBuildBr(b, opblocks[opno + 1]);
break;
+ case EEOP_RETURNINGEXPR:
+ {
+ LLVMBasicBlockRef b_isnull;
+ LLVMValueRef v_flagsp;
+ LLVMValueRef v_flags;
+ LLVMValueRef v_nullflag;
+
+ b_isnull = l_bb_before_v(opblocks[opno + 1],
+ "op.%d.row.isnull", opno);
+
+ /*
+ * The next op actually evaluates the expression. If the
+ * OLD/NEW row doesn't exist, skip that and return NULL.
+ */
+ v_flagsp = l_struct_gep(b,
+ StructExprState,
+ v_state,
+ FIELDNO_EXPRSTATE_FLAGS,
+ "v.state.flags");
+ v_flags = l_load(b, TypeStorageBool, v_flagsp, "");
+
+ v_nullflag = l_int8_const(lc, op->d.returningexpr.nullflag);
+
+ LLVMBuildCondBr(b,
+ LLVMBuildICmp(b, LLVMIntEQ,
+ LLVMBuildAnd(b, v_flags,
+ v_nullflag, ""),
+ l_sbool_const(0), ""),
+ opblocks[opno + 1], b_isnull);
+
+ LLVMPositionBuilderAtEnd(b, b_isnull);
+
+ LLVMBuildStore(b, l_sizet_const(0), v_resvaluep);
+ LLVMBuildStore(b, l_sbool_const(1), v_resnullp);
+
+ LLVMBuildBr(b, opblocks[op->d.returningexpr.jumpdone]);
+ break;
+ }
+
case EEOP_ARRAYEXPR:
build_EvalXFunc(b, mod, "ExecEvalArrayExpr",
v_state, op);
diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c
new file mode 100644
index 9cac3c1..4e25ca6
--- a/src/backend/nodes/makefuncs.c
+++ b/src/backend/nodes/makefuncs.c
@@ -80,12 +80,14 @@ makeVar(int varno,
var->varlevelsup = varlevelsup;
/*
- * Only a few callers need to make Var nodes with non-null varnullingrels,
- * or with varnosyn/varattnosyn different from varno/varattno. We don't
- * provide separate arguments for them, but just initialize them to NULL
- * and the given varno/varattno. This reduces code clutter and chance of
- * error for most callers.
+ * Only a few callers need to make Var nodes with varreturningtype
+ * different from VAR_RETURNING_DEFAULT, non-null varnullingrels, or with
+ * varnosyn/varattnosyn different from varno/varattno. We don't provide
+ * separate arguments for them, but just initialize them to sensible
+ * default values. This reduces code clutter and chance of error for most
+ * callers.
*/
+ var->varreturningtype = VAR_RETURNING_DEFAULT;
var->varnullingrels = NULL;
var->varnosyn = (Index) varno;
var->varattnosyn = varattno;
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
new file mode 100644
index f760722..afd6296
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -278,6 +278,9 @@ exprType(const Node *expr)
type = exprType((Node *) n->expr);
}
break;
+ case T_ReturningExpr:
+ type = exprType((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
type = exprType((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -529,6 +532,8 @@ exprTypmod(const Node *expr)
return ((const CoerceToDomainValue *) expr)->typeMod;
case T_SetToDefault:
return ((const SetToDefault *) expr)->typeMod;
+ case T_ReturningExpr:
+ return exprTypmod((Node *) ((const ReturningExpr *) expr)->retexpr);
case T_PlaceHolderVar:
return exprTypmod((Node *) ((const PlaceHolderVar *) expr)->phexpr);
default:
@@ -1047,6 +1052,9 @@ exprCollation(const Node *expr)
case T_InferenceElem:
coll = exprCollation((Node *) ((const InferenceElem *) expr)->expr);
break;
+ case T_ReturningExpr:
+ coll = exprCollation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
coll = exprCollation((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -1298,6 +1306,10 @@ exprSetCollation(Node *expr, Oid collati
/* NextValueExpr's result is an integer type ... */
Assert(!OidIsValid(collation)); /* ... so never set a collation */
break;
+ case T_ReturningExpr:
+ exprSetCollation((Node *) ((ReturningExpr *) expr)->retexpr,
+ collation);
+ break;
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(expr));
break;
@@ -1624,6 +1636,9 @@ exprLocation(const Node *expr)
case T_SetToDefault:
loc = ((const SetToDefault *) expr)->location;
break;
+ case T_ReturningExpr:
+ loc = exprLocation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_TargetEntry:
/* just use argument's location */
loc = exprLocation((Node *) ((const TargetEntry *) expr)->expr);
@@ -2614,6 +2629,8 @@ expression_tree_walker_impl(Node *node,
return WALK(((PlaceHolderVar *) node)->phexpr);
case T_InferenceElem:
return WALK(((InferenceElem *) node)->expr);
+ case T_ReturningExpr:
+ return WALK(((ReturningExpr *) node)->retexpr);
case T_AppendRelInfo:
{
AppendRelInfo *appinfo = (AppendRelInfo *) node;
@@ -3455,6 +3472,16 @@ expression_tree_mutator_impl(Node *node,
return (Node *) newnode;
}
break;
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ ReturningExpr *newnode;
+
+ FLATCOPY(newnode, rexpr, ReturningExpr);
+ MUTATE(newnode->retexpr, rexpr->retexpr, Expr *);
+ return (Node *) newnode;
+ }
+ break;
case T_TargetEntry:
{
TargetEntry *targetentry = (TargetEntry *) node;
@@ -4006,6 +4033,7 @@ raw_expression_tree_walker_impl(Node *no
case T_A_Const:
case T_A_Star:
case T_MergeSupportFunc:
+ case T_ReturningOption:
/* primitive node types with no subnodes */
break;
case T_Alias:
@@ -4234,7 +4262,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->onConflictClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4250,7 +4278,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->whereClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4268,7 +4296,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->fromClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4286,7 +4314,7 @@ raw_expression_tree_walker_impl(Node *no
return true;
if (WALK(stmt->mergeWhenClauses))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4304,6 +4332,16 @@ raw_expression_tree_walker_impl(Node *no
return true;
}
break;
+ case T_ReturningClause:
+ {
+ ReturningClause *returning = (ReturningClause *) node;
+
+ if (WALK(returning->options))
+ return true;
+ if (WALK(returning->exprs))
+ return true;
+ }
+ break;
case T_SelectStmt:
{
SelectStmt *stmt = (SelectStmt *) node;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
new file mode 100644
index 172edb6..6346c4e
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -3985,6 +3985,7 @@ subquery_push_qual(Query *subquery, Rang
*/
qual = ReplaceVarsFromTargetList(qual, rti, 0, rte,
subquery->targetList,
+ subquery->resultRelation,
REPLACEVARS_REPORT_ERROR, 0,
&subquery->hasSubLinks);
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
new file mode 100644
index f2ed0d8..108c055
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -7114,6 +7114,8 @@ make_modifytable(PlannerInfo *root, Plan
int epqParam)
{
ModifyTable *node = makeNode(ModifyTable);
+ bool returning_old_or_new = false;
+ bool returning_old_or_new_valid = false;
List *fdw_private_list;
Bitmapset *direct_modify_plans;
ListCell *lc;
@@ -7178,6 +7180,8 @@ make_modifytable(PlannerInfo *root, Plan
}
node->updateColnosLists = updateColnosLists;
node->withCheckOptionLists = withCheckOptionLists;
+ node->returningOld = root->parse->returningOld;
+ node->returningNew = root->parse->returningNew;
node->returningLists = returningLists;
node->rowMarks = rowMarks;
node->mergeActionLists = mergeActionLists;
@@ -7258,7 +7262,8 @@ make_modifytable(PlannerInfo *root, Plan
* callback functions needed for that and (2) there are no local
* structures that need to be run for each modified row: row-level
* triggers on the foreign table, stored generated columns, WITH CHECK
- * OPTIONs from parent views.
+ * OPTIONs from parent views, or Vars returning OLD/NEW in the
+ * RETURNING list.
*/
direct_modify = false;
if (fdwroutine != NULL &&
@@ -7269,7 +7274,18 @@ make_modifytable(PlannerInfo *root, Plan
withCheckOptionLists == NIL &&
!has_row_triggers(root, rti, operation) &&
!has_stored_generated_columns(root, rti))
- direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ {
+ /* returning_old_or_new is the same for all result relations */
+ if (!returning_old_or_new_valid)
+ {
+ returning_old_or_new =
+ contain_vars_returning_old_or_new((Node *)
+ root->parse->returningList);
+ returning_old_or_new_valid = true;
+ }
+ if (!returning_old_or_new)
+ direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ }
if (direct_modify)
direct_modify_plans = bms_add_member(direct_modify_plans, i);
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
new file mode 100644
index 91c7c4f..218e46a
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -3074,6 +3074,21 @@ fix_join_expr_mutator(Node *node, fix_jo
{
Var *var = (Var *) node;
+ /*
+ * Verify that Vars with non-default varreturningtype only appear in
+ * the RETURNING list, and refer to the target relation.
+ */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ if (context->inner_itlist != NULL ||
+ context->outer_itlist == NULL ||
+ context->acceptable_rel == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+ if (var->varno != context->acceptable_rel)
+ elog(ERROR, "wrong varno %d (expected %d) for variable returning old/new",
+ var->varno, context->acceptable_rel);
+ }
+
/* Look for the var in the input tlists, first in the outer */
if (context->outer_itlist)
{
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
new file mode 100644
index 09d5f0f..86d215c
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -354,17 +354,19 @@ build_subplan(PlannerInfo *root, Plan *p
Node *arg = pitem->item;
/*
- * The Var, PlaceHolderVar, Aggref or GroupingFunc has already been
- * adjusted to have the correct varlevelsup, phlevelsup, or
- * agglevelsup.
+ * The Var, PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr has
+ * already been adjusted to have the correct varlevelsup, phlevelsup,
+ * agglevelsup, or retlevelsup.
*
- * If it's a PlaceHolderVar, Aggref or GroupingFunc, its arguments
- * might contain SubLinks, which have not yet been processed (see the
- * comments for SS_replace_correlation_vars). Do that now.
+ * If it's a PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr,
+ * its arguments might contain SubLinks, which have not yet been
+ * processed (see the comments for SS_replace_correlation_vars). Do
+ * that now.
*/
if (IsA(arg, PlaceHolderVar) ||
IsA(arg, Aggref) ||
- IsA(arg, GroupingFunc))
+ IsA(arg, GroupingFunc) ||
+ IsA(arg, ReturningExpr))
arg = SS_process_sublinks(root, arg, false);
splan->parParam = lappend_int(splan->parParam, pitem->paramId);
@@ -1866,8 +1868,8 @@ convert_EXISTS_to_ANY(PlannerInfo *root,
/*
* Replace correlation vars (uplevel vars) with Params.
*
- * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions, and
- * MergeSupportFuncs are replaced, too.
+ * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions,
+ * MergeSupportFuncs, and ReturningExprs are replaced, too.
*
* Note: it is critical that this runs immediately after SS_process_sublinks.
* Since we do not recurse into the arguments of uplevel PHVs and aggregates,
@@ -1927,6 +1929,12 @@ replace_correlation_vars_mutator(Node *n
return (Node *) replace_outer_merge_support(root,
(MergeSupportFunc *) node);
}
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return (Node *) replace_outer_returning(root,
+ (ReturningExpr *) node);
+ }
return expression_tree_mutator(node,
replace_correlation_vars_mutator,
(void *) root);
@@ -1982,11 +1990,11 @@ process_sublinks_mutator(Node *node, pro
}
/*
- * Don't recurse into the arguments of an outer PHV, Aggref or
- * GroupingFunc here. Any SubLinks in the arguments have to be dealt with
- * at the outer query level; they'll be handled when build_subplan
- * collects the PHV, Aggref or GroupingFunc into the arguments to be
- * passed down to the current subplan.
+ * Don't recurse into the arguments of an outer PHV, Aggref, GroupingFunc
+ * or ReturningExpr here. Any SubLinks in the arguments have to be dealt
+ * with at the outer query level; they'll be handled when build_subplan
+ * collects the PHV, Aggref, GroupingFunc or ReturningExpr into the
+ * arguments to be passed down to the current subplan.
*/
if (IsA(node, PlaceHolderVar))
{
@@ -2003,6 +2011,11 @@ process_sublinks_mutator(Node *node, pro
if (((GroupingFunc *) node)->agglevelsup > 0)
return node;
}
+ else if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return node;
+ }
/*
* We should never see a SubPlan expression in the input (since this is
@@ -2115,7 +2128,9 @@ SS_identify_outer_params(PlannerInfo *ro
outer_params = NULL;
for (proot = root->parent_root; proot != NULL; proot = proot->parent_root)
{
- /* Include ordinary Var/PHV/Aggref/GroupingFunc params */
+ /*
+ * Include ordinary Var/PHV/Aggref/GroupingFunc/ReturningExpr params.
+ */
foreach(l, proot->plan_params)
{
PlannerParamItem *pitem = (PlannerParamItem *) lfirst(l);
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
new file mode 100644
index 4d7f972..79d3e99
--- a/src/backend/optimizer/prep/prepjointree.c
+++ b/src/backend/optimizer/prep/prepjointree.c
@@ -2512,7 +2512,8 @@ pullup_replace_vars_callback(Var *var,
* expansion with varlevelsup = 0, and then adjust below if needed.
*/
expandRTE(rcon->target_rte,
- var->varno, 0 /* not varlevelsup */ , var->location,
+ var->varno, 0 /* not varlevelsup */ ,
+ var->varreturningtype, var->location,
(var->vartype != RECORDOID),
&colnames, &fields);
/* Expand the generated per-field Vars, but don't insert PHVs there */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
new file mode 100644
index 4989722..7a6fe58
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -253,6 +253,13 @@ adjust_appendrel_attrs_mutator(Node *nod
* all non-Var outputs of such subqueries, and then we could look up
* the pre-existing PHV here. Or perhaps just wrap the translations
* that way to begin with?
+ *
+ * If var->varreturningtype is not VAR_RETURNING_DEFAULT, then that
+ * also needs to be copied to the translated Var. That too would fail
+ * if the translation wasn't a Var, but that should never happen since
+ * a non-default var->varreturningtype is only used for Vars referring
+ * to the result relation, which should never be a flattened UNION ALL
+ * subquery.
*/
for (cnt = 0; cnt < nappinfos; cnt++)
@@ -283,9 +290,17 @@ adjust_appendrel_attrs_mutator(Node *nod
elog(ERROR, "attribute %d of relation \"%s\" does not exist",
var->varattno, get_rel_name(appinfo->parent_reloid));
if (IsA(newnode, Var))
+ {
+ ((Var *) newnode)->varreturningtype = var->varreturningtype;
((Var *) newnode)->varnullingrels = var->varnullingrels;
- else if (var->varnullingrels != NULL)
- elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
+ else
+ {
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
+ if (var->varnullingrels != NULL)
+ elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
return newnode;
}
else if (var->varattno == 0)
@@ -339,6 +354,8 @@ adjust_appendrel_attrs_mutator(Node *nod
rowexpr->colnames = copyObject(rte->eref->colnames);
rowexpr->location = -1;
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
if (var->varnullingrels != NULL)
elog(ERROR, "failed to apply nullingrels to a non-Var");
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
new file mode 100644
index 8e39795..8f193f3
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -1296,6 +1296,7 @@ contain_leaked_vars_walker(Node *node, v
case T_NullTest:
case T_BooleanTest:
case T_NextValueExpr:
+ case T_ReturningExpr:
case T_List:
/*
@@ -3405,6 +3406,8 @@ eval_const_expressions_mutator(Node *nod
fselect->resulttypmod,
fselect->resultcollid,
((Var *) arg)->varlevelsup);
+ /* New Var has same OLD/NEW returning as old one */
+ newvar->varreturningtype = ((Var *) arg)->varreturningtype;
/* New Var is nullable by same rels as the old one */
newvar->varnullingrels = ((Var *) arg)->varnullingrels;
return (Node *) newvar;
diff --git a/src/backend/optimizer/util/paramassign.c b/src/backend/optimizer/util/paramassign.c
new file mode 100644
index f461fed..38a3986
--- a/src/backend/optimizer/util/paramassign.c
+++ b/src/backend/optimizer/util/paramassign.c
@@ -91,6 +91,7 @@ assign_param_for_var(PlannerInfo *root,
pvar->vartype == var->vartype &&
pvar->vartypmod == var->vartypmod &&
pvar->varcollid == var->varcollid &&
+ pvar->varreturningtype == var->varreturningtype &&
bms_equal(pvar->varnullingrels, var->varnullingrels))
return pitem->paramId;
}
@@ -357,6 +358,52 @@ replace_outer_merge_support(PlannerInfo
return retval;
}
+
+/*
+ * Generate a Param node to replace the given ReturningExpr expression which
+ * is expected to have retlevelsup > 0 (ie, it is not local). Record the need
+ * for the ReturningExpr in the proper upper-level root->plan_params.
+ */
+Param *
+replace_outer_returning(PlannerInfo *root, ReturningExpr *rexpr)
+{
+ Param *retval;
+ PlannerParamItem *pitem;
+ Index levelsup;
+ Oid ptype = exprType((Node *) rexpr->retexpr);
+
+ Assert(rexpr->retlevelsup > 0 && rexpr->retlevelsup < root->query_level);
+
+ /* Find the query level the ReturningExpr belongs to */
+ for (levelsup = rexpr->retlevelsup; levelsup > 0; levelsup--)
+ root = root->parent_root;
+
+ /*
+ * It does not seem worthwhile to try to de-duplicate references to outer
+ * ReturningExprs. Just make a new slot every time.
+ */
+ rexpr = copyObject(rexpr);
+ IncrementVarSublevelsUp((Node *) rexpr, -((int) rexpr->retlevelsup), 0);
+ Assert(rexpr->retlevelsup == 0);
+
+ pitem = makeNode(PlannerParamItem);
+ pitem->item = (Node *) rexpr;
+ pitem->paramId = list_length(root->glob->paramExecTypes);
+ root->glob->paramExecTypes = lappend_oid(root->glob->paramExecTypes,
+ ptype);
+
+ root->plan_params = lappend(root->plan_params, pitem);
+
+ retval = makeNode(Param);
+ retval->paramkind = PARAM_EXEC;
+ retval->paramid = pitem->paramId;
+ retval->paramtype = ptype;
+ retval->paramtypmod = exprTypmod((Node *) rexpr->retexpr);
+ retval->paramcollid = exprCollation((Node *) rexpr->retexpr);
+ retval->location = exprLocation((Node *) rexpr->retexpr);
+
+ return retval;
+}
/*
* Generate a Param node to replace the given Var,
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
new file mode 100644
index b913f91..16e6353
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1843,8 +1843,8 @@ build_physical_tlist(PlannerInfo *root,
case RTE_NAMEDTUPLESTORE:
case RTE_RESULT:
/* Not all of these can have dropped cols, but share code anyway */
- expandRTE(rte, varno, 0, -1, true /* include dropped */ ,
- NULL, &colvars);
+ expandRTE(rte, varno, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , NULL, &colvars);
foreach(l, colvars)
{
var = (Var *) lfirst(l);
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
new file mode 100644
index f7534ad..4b50767
--- a/src/backend/optimizer/util/var.c
+++ b/src/backend/optimizer/util/var.c
@@ -76,6 +76,7 @@ static bool pull_varattnos_walker(Node *
static bool pull_vars_walker(Node *node, pull_vars_context *context);
static bool contain_var_clause_walker(Node *node, void *context);
static bool contain_vars_of_level_walker(Node *node, int *sublevels_up);
+static bool contain_vars_returning_old_or_new_walker(Node *node, void *context);
static bool locate_var_of_level_walker(Node *node,
locate_var_of_level_context *context);
static bool pull_var_clause_walker(Node *node,
@@ -495,6 +496,49 @@ contain_vars_of_level_walker(Node *node,
}
+/*
+ * contain_vars_returning_old_or_new
+ * Recursively scan a clause to discover whether it contains any Var nodes
+ * (of the current query level) whose varreturningtype is VAR_RETURNING_OLD
+ * or VAR_RETURNING_NEW.
+ *
+ * Returns true if any found.
+ *
+ * Any ReturningExprs are also detected --- if an OLD/NEW Var was rewritten,
+ * we still regard this as a clause that returns OLD/NEW values.
+ *
+ * Does not examine subqueries, therefore must only be used after reduction
+ * of sublinks to subplans!
+ */
+bool
+contain_vars_returning_old_or_new(Node *node)
+{
+ return contain_vars_returning_old_or_new_walker(node, NULL);
+}
+
+static bool
+contain_vars_returning_old_or_new_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ if (((Var *) node)->varlevelsup == 0 &&
+ ((Var *) node)->varreturningtype != VAR_RETURNING_DEFAULT)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup == 0)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ return expression_tree_walker(node, contain_vars_returning_old_or_new_walker,
+ context);
+}
+
+
/*
* locate_var_of_level
* Find the parse location of any Var of the specified query level.
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
new file mode 100644
index 506e063..372cf51
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -641,8 +641,8 @@ transformDeleteStmt(ParseState *pstate,
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -1054,7 +1054,7 @@ transformInsertStmt(ParseState *pstate,
* contain only the target relation, removing any entries added in a
* sub-SELECT or VALUES list.
*/
- if (stmt->onConflictClause || stmt->returningList)
+ if (stmt->onConflictClause || stmt->returningClause)
{
pstate->p_namespace = NIL;
addNSItemToQuery(pstate, pstate->p_target_nsitem,
@@ -1067,10 +1067,9 @@ transformInsertStmt(ParseState *pstate,
stmt->onConflictClause);
/* Process RETURNING, if any. */
- if (stmt->returningList)
- qry->returningList = transformReturningList(pstate,
- stmt->returningList,
- EXPR_KIND_RETURNING);
+ if (stmt->returningClause)
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -2548,8 +2547,8 @@ transformUpdateStmt(ParseState *pstate,
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/*
* Now we are done with SELECT-like processing, and can get on with
@@ -2645,18 +2644,120 @@ transformUpdateTargetList(ParseState *ps
}
/*
- * transformReturningList -
+ * addNSItemForReturning -
+ * add a ParseNamespaceItem for the OLD or NEW alias in RETURNING.
+ */
+static void
+addNSItemForReturning(ParseState *pstate, const char *aliasname,
+ VarReturningType returning_type)
+{
+ List *colnames;
+ int numattrs;
+ ParseNamespaceColumn *nscolumns;
+ ParseNamespaceItem *nsitem;
+
+ /* copy per-column data from the target relation */
+ colnames = pstate->p_target_nsitem->p_rte->eref->colnames;
+ numattrs = list_length(colnames);
+
+ nscolumns = (ParseNamespaceColumn *)
+ palloc(numattrs * sizeof(ParseNamespaceColumn));
+
+ memcpy(nscolumns, pstate->p_target_nsitem->p_nscolumns,
+ numattrs * sizeof(ParseNamespaceColumn));
+
+ /* mark all columns as returning OLD/NEW */
+ for (int i = 0; i < numattrs; i++)
+ nscolumns[i].p_varreturningtype = returning_type;
+
+ /* build the nsitem, copying most fields from the target relation */
+ nsitem = (ParseNamespaceItem *) palloc(sizeof(ParseNamespaceItem));
+ nsitem->p_names = makeAlias(aliasname, colnames);
+ nsitem->p_rte = pstate->p_target_nsitem->p_rte;
+ nsitem->p_rtindex = pstate->p_target_nsitem->p_rtindex;
+ nsitem->p_perminfo = pstate->p_target_nsitem->p_perminfo;
+ nsitem->p_nscolumns = nscolumns;
+ nsitem->p_returning_type = returning_type;
+
+ /* add it to the query namespace as a table-only item */
+ addNSItemToQuery(pstate, nsitem, false, true, false);
+}
+
+/*
+ * transformReturningClause -
* handle a RETURNING clause in INSERT/UPDATE/DELETE/MERGE
*/
-List *
-transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind)
+void
+transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind)
{
- List *rlist;
+ int save_nslen;
int save_next_resno;
- if (returningList == NIL)
- return NIL; /* nothing to do */
+ if (returningClause == NULL)
+ return; /* nothing to do */
+
+ /*
+ * Scan RETURNING WITH(...) options for OLD/NEW alias names. Complain if
+ * there is any conflict with existing relations.
+ */
+ foreach_node(ReturningOption, option, returningClause->options)
+ {
+ switch (option->option)
+ {
+ case RETURNING_OPTION_OLD:
+ if (qry->returningOld != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "OLD"),
+ parser_errposition(pstate, option->location));
+ qry->returningOld = option->value;
+ break;
+
+ case RETURNING_OPTION_NEW:
+ if (qry->returningNew != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "NEW"),
+ parser_errposition(pstate, option->location));
+ qry->returningNew = option->value;
+ break;
+
+ default:
+ elog(ERROR, "unrecognized returning option: %d", option->option);
+ }
+
+ if (refnameNamespaceItem(pstate, NULL, option->value, -1, NULL) != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_ALIAS),
+ errmsg("table name \"%s\" specified more than once",
+ option->value),
+ parser_errposition(pstate, option->location));
+ }
+
+ /*
+ * If no OLD/NEW aliases specified, use "old"/"new" unless masked by
+ * existing relations.
+ */
+ if (qry->returningOld == NULL &&
+ refnameNamespaceItem(pstate, NULL, "old", -1, NULL) == NULL)
+ qry->returningOld = "old";
+ if (qry->returningNew == NULL &&
+ refnameNamespaceItem(pstate, NULL, "new", -1, NULL) == NULL)
+ qry->returningNew = "new";
+
+ /*
+ * Add the OLD and NEW aliases to the query namespace, for use in
+ * expressions in the RETURNING list.
+ */
+ save_nslen = list_length(pstate->p_namespace);
+ if (qry->returningOld != NULL)
+ addNSItemForReturning(pstate, qry->returningOld, VAR_RETURNING_OLD);
+ if (qry->returningNew != NULL)
+ addNSItemForReturning(pstate, qry->returningNew, VAR_RETURNING_NEW);
/*
* We need to assign resnos starting at one in the RETURNING list. Save
@@ -2666,8 +2767,10 @@ transformReturningList(ParseState *pstat
save_next_resno = pstate->p_next_resno;
pstate->p_next_resno = 1;
- /* transform RETURNING identically to a SELECT targetlist */
- rlist = transformTargetList(pstate, returningList, exprKind);
+ /* transform RETURNING expressions identically to a SELECT targetlist */
+ qry->returningList = transformTargetList(pstate,
+ returningClause->exprs,
+ exprKind);
/*
* Complain if the nonempty tlist expanded to nothing (which is possible
@@ -2675,24 +2778,23 @@ transformReturningList(ParseState *pstat
* allow this, the parsed Query will look like it didn't have RETURNING,
* with results that would probably surprise the user.
*/
- if (rlist == NIL)
+ if (qry->returningList == NIL)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("RETURNING must have at least one column"),
parser_errposition(pstate,
- exprLocation(linitial(returningList)))));
+ exprLocation(linitial(returningClause->exprs)))));
/* mark column origins */
- markTargetListOrigins(pstate, rlist);
+ markTargetListOrigins(pstate, qry->returningList);
/* resolve any still-unresolved output columns as being type text */
if (pstate->p_resolve_unknowns)
- resolveTargetListUnknowns(pstate, rlist);
+ resolveTargetListUnknowns(pstate, qry->returningList);
/* restore state */
+ pstate->p_namespace = list_truncate(pstate->p_namespace, save_nslen);
pstate->p_next_resno = save_next_resno;
-
- return rlist;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
new file mode 100644
index dd45818..3d9a055
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -261,6 +261,8 @@ static Node *makeRecursiveViewSelect(cha
MergeWhenClause *mergewhen;
struct KeyActions *keyactions;
struct KeyAction *keyaction;
+ ReturningClause *retclause;
+ ReturningOptionKind retoptionkind;
}
%type <node> stmt toplevel_stmt schema_stmt routine_body_stmt
@@ -430,7 +432,8 @@ static Node *makeRecursiveViewSelect(cha
opclass_purpose opt_opfamily transaction_mode_list_or_empty
OptTableFuncElementList TableFuncElementList opt_type_modifiers
prep_type_clause
- execute_param_clause using_clause returning_clause
+ execute_param_clause using_clause
+ returning_with_clause returning_options
opt_enum_val_list enum_val_list table_func_column_list
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
@@ -439,6 +442,9 @@ static Node *makeRecursiveViewSelect(cha
vacuum_relation_list opt_vacuum_relation_list
drop_option_list pub_obj_list
+%type <retclause> returning_clause
+%type <node> returning_option
+%type <retoptionkind> returning_option_kind
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
@@ -12146,7 +12152,7 @@ InsertStmt:
{
$5->relation = $4;
$5->onConflictClause = $6;
- $5->returningList = $7;
+ $5->returningClause = $7;
$5->withClause = $1;
$5->stmt_location = @$;
$$ = (Node *) $5;
@@ -12280,8 +12286,45 @@ opt_conf_expr:
;
returning_clause:
- RETURNING target_list { $$ = $2; }
- | /* EMPTY */ { $$ = NIL; }
+ RETURNING returning_with_clause target_list
+ {
+ ReturningClause *n = makeNode(ReturningClause);
+
+ n->options = $2;
+ n->exprs = $3;
+ $$ = n;
+ }
+ | /* EMPTY */
+ {
+ $$ = NULL;
+ }
+ ;
+
+returning_with_clause:
+ WITH '(' returning_options ')' { $$ = $3; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+returning_options:
+ returning_option { $$ = list_make1($1); }
+ | returning_options ',' returning_option { $$ = lappend($1, $3); }
+ ;
+
+returning_option:
+ returning_option_kind AS ColId
+ {
+ ReturningOption *n = makeNode(ReturningOption);
+
+ n->option = $1;
+ n->value = $3;
+ n->location = @1;
+ $$ = (Node *) n;
+ }
+ ;
+
+returning_option_kind:
+ OLD { $$ = RETURNING_OPTION_OLD; }
+ | NEW { $$ = RETURNING_OPTION_NEW; }
;
@@ -12300,7 +12343,7 @@ DeleteStmt: opt_with_clause DELETE_P FRO
n->relation = $4;
n->usingClause = $5;
n->whereClause = $6;
- n->returningList = $7;
+ n->returningClause = $7;
n->withClause = $1;
n->stmt_location = @$;
$$ = (Node *) n;
@@ -12375,7 +12418,7 @@ UpdateStmt: opt_with_clause UPDATE relat
n->targetList = $5;
n->fromClause = $6;
n->whereClause = $7;
- n->returningList = $8;
+ n->returningClause = $8;
n->withClause = $1;
n->stmt_location = @$;
$$ = (Node *) n;
@@ -12454,7 +12497,7 @@ MergeStmt:
m->sourceRelation = $6;
m->joinCondition = $8;
m->mergeWhenClauses = $9;
- m->returningList = $10;
+ m->returningClause = $10;
m->stmt_location = @$;
$$ = (Node *) m;
diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c
new file mode 100644
index 979926b..725591f
--- a/src/backend/parser/parse_clause.c
+++ b/src/backend/parser/parse_clause.c
@@ -1585,6 +1585,7 @@ transformFromClauseItem(ParseState *psta
jnsitem->p_cols_visible = true;
jnsitem->p_lateral_only = false;
jnsitem->p_lateral_ok = true;
+ jnsitem->p_returning_type = VAR_RETURNING_DEFAULT;
/* Per SQL, we must check for alias conflicts */
checkNameSpaceConflicts(pstate, list_make1(jnsitem), my_namespace);
my_namespace = lappend(my_namespace, jnsitem);
@@ -1647,6 +1648,7 @@ buildVarFromNSColumn(ParseState *pstate,
nscol->p_varcollid,
0);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
new file mode 100644
index c280629..18769a6
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -2619,6 +2619,13 @@ transformWholeRowRef(ParseState *pstate,
* point, there seems no harm in expanding it now rather than during
* planning.
*
+ * Note that if the nsitem is an OLD/NEW alias for the target RTE (as can
+ * appear in a RETURNING list), its alias won't match the target RTE's
+ * alias, but we still want to make a whole-row Var here rather than a
+ * RowExpr, for consistency with direct references to the target RTE, and
+ * so that any dropped columns are handled correctly. Thus we also check
+ * p_returning_type here.
+ *
* Note that if the RTE is a function returning scalar, we create just a
* plain reference to the function value, not a composite containing a
* single column. This is pretty inconsistent at first sight, but it's
@@ -2626,13 +2633,17 @@ transformWholeRowRef(ParseState *pstate,
* "rel.*" mean the same thing for composite relations, so why not for
* scalar functions...
*/
- if (nsitem->p_names == nsitem->p_rte->eref)
+ if (nsitem->p_names == nsitem->p_rte->eref ||
+ nsitem->p_returning_type != VAR_RETURNING_DEFAULT)
{
Var *result;
result = makeWholeRowVar(nsitem->p_rte, nsitem->p_rtindex,
sublevels_up, true);
+ /* mark Var for RETURNING OLD/NEW, as necessary */
+ result->varreturningtype = nsitem->p_returning_type;
+
/* location is not filled in by makeWholeRowVar */
result->location = location;
@@ -2655,9 +2666,8 @@ transformWholeRowRef(ParseState *pstate,
* are in the RTE. We needn't worry about marking the RTE for SELECT
* access, as the common columns are surely so marked already.
*/
- expandRTE(nsitem->p_rte, nsitem->p_rtindex,
- sublevels_up, location, false,
- NULL, &fields);
+ expandRTE(nsitem->p_rte, nsitem->p_rtindex, sublevels_up,
+ nsitem->p_returning_type, location, false, NULL, &fields);
rowexpr = makeNode(RowExpr);
rowexpr->args = list_truncate(fields,
list_length(nsitem->p_names->colnames));
diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c
new file mode 100644
index 87df790..0eb8bb4
--- a/src/backend/parser/parse_merge.c
+++ b/src/backend/parser/parse_merge.c
@@ -247,8 +247,8 @@ transformMergeStmt(ParseState *pstate, M
qry->jointree = makeFromExpr(pstate->p_joinlist, NULL);
/* Transform the RETURNING list, if any */
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_MERGE_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_MERGE_RETURNING);
/*
* We now have a good query shape, so now look at the WHEN conditions and
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
new file mode 100644
index 8075b1b..610d879
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -91,11 +91,13 @@ static void markRTEForSelectPriv(ParseSt
int rtindex, AttrNumber col);
static void expandRelation(Oid relid, Alias *eref,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static int specialAttNum(const char *attname);
@@ -763,6 +765,9 @@ scanNSItemForColumn(ParseState *pstate,
}
var->location = location;
+ /* Mark Var for RETURNING OLD/NEW, as necessary */
+ var->varreturningtype = nsitem->p_returning_type;
+
/* Mark Var if it's nulled by any outer joins */
markNullableIfNeeded(pstate, var);
@@ -1336,6 +1341,7 @@ buildNSItemFromTupleDesc(RangeTblEntry *
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -1399,6 +1405,7 @@ buildNSItemFromLists(RangeTblEntry *rte,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2300,6 +2307,7 @@ addRangeTableEntryForJoin(ParseState *ps
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2720,9 +2728,10 @@ addNSItemToQuery(ParseState *pstate, Par
* results. If include_dropped is true then empty strings and NULL constants
* (not Vars!) are returned for dropped columns.
*
- * rtindex, sublevels_up, and location are the varno, varlevelsup, and location
- * values to use in the created Vars. Ordinarily rtindex should match the
- * actual position of the RTE in its rangetable.
+ * rtindex, sublevels_up, returning_type, and location are the varno,
+ * varlevelsup, varreturningtype, and location values to use in the created
+ * Vars. Ordinarily rtindex should match the actual position of the RTE in
+ * its rangetable.
*
* The output lists go into *colnames and *colvars.
* If only one of the two kinds of output list is needed, pass NULL for the
@@ -2730,6 +2739,7 @@ addNSItemToQuery(ParseState *pstate, Par
*/
void
expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -2745,7 +2755,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
case RTE_RELATION:
/* Ordinary relation RTE */
expandRelation(rte->relid, rte->eref,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up, returning_type, location,
include_dropped, colnames, colvars);
break;
case RTE_SUBQUERY:
@@ -2792,6 +2802,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
exprTypmod((Node *) te->expr),
exprCollation((Node *) te->expr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2829,7 +2840,8 @@ expandRTE(RangeTblEntry *rte, int rtinde
Assert(tupdesc);
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up,
+ returning_type, location,
include_dropped, colnames, colvars);
}
else if (functypclass == TYPEFUNC_SCALAR)
@@ -2849,6 +2861,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
exprTypmod(rtfunc->funcexpr),
exprCollation(rtfunc->funcexpr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2891,6 +2904,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
attrtypmod,
attrcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
}
@@ -2920,6 +2934,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
InvalidOid,
sublevels_up);
+ varnode->varreturningtype = returning_type;
*colvars = lappend(*colvars, varnode);
}
}
@@ -3002,6 +3017,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
exprTypmod(avar),
exprCollation(avar),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3057,6 +3073,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
varnode = makeVar(rtindex, varattno,
coltype, coltypmod, colcoll,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3089,6 +3106,7 @@ expandRTE(RangeTblEntry *rte, int rtinde
*/
static void
expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3097,7 +3115,7 @@ expandRelation(Oid relid, Alias *eref, i
/* Get the tupledesc and turn it over to expandTupleDesc */
rel = relation_open(relid, AccessShareLock);
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
- rtindex, sublevels_up,
+ rtindex, sublevels_up, returning_type,
location, include_dropped,
colnames, colvars);
relation_close(rel, AccessShareLock);
@@ -3115,6 +3133,7 @@ expandRelation(Oid relid, Alias *eref, i
static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3175,6 +3194,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias
attr->atttypid, attr->atttypmod,
attr->attcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3227,6 +3247,7 @@ expandNSItemVars(ParseState *pstate, Par
nscol->p_varcollid,
sublevels_up);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
var->location = location;
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
new file mode 100644
index 76bf88c..f90afe2
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -1550,8 +1550,8 @@ expandRecordVariable(ParseState *pstate,
*lvar;
int i;
- expandRTE(rte, var->varno, 0, var->location, false,
- &names, &vars);
+ expandRTE(rte, var->varno, 0, var->varreturningtype,
+ var->location, false, &names, &vars);
tupleDesc = CreateTemplateTupleDesc(list_length(vars));
i = 1;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
new file mode 100644
index 6d59a2b..e8b86e2
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -635,6 +635,7 @@ rewriteRuleAction(Query *parsetree,
0,
rt_fetch(new_varno, sub_action->rtable),
parsetree->targetList,
+ sub_action->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -668,10 +669,15 @@ rewriteRuleAction(Query *parsetree,
rt_fetch(parsetree->resultRelation,
parsetree->rtable),
rule_action->returningList,
+ rule_action->resultRelation,
REPLACEVARS_REPORT_ERROR,
0,
&rule_action->hasSubLinks);
+ /* use triggering query's aliases for OLD and NEW in RETURNING list */
+ rule_action->returningOld = parsetree->returningOld;
+ rule_action->returningNew = parsetree->returningNew;
+
/*
* There could have been some SubLinks in parsetree's returningList,
* in which case we'd better mark the rule_action correctly.
@@ -2304,6 +2310,7 @@ CopyAndAddInvertedQual(Query *parsetree,
rt_fetch(rt_index,
parsetree->rtable),
parsetree->targetList,
+ parsetree->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -3528,6 +3535,7 @@ rewriteTargetView(Query *parsetree, Rela
0,
view_rte,
view_targetlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
NULL);
@@ -3679,6 +3687,7 @@ rewriteTargetView(Query *parsetree, Rela
0,
view_rte,
tmp_tlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
&parsetree->hasSubLinks);
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
new file mode 100644
index 8f90afb..0546667
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -817,6 +817,14 @@ IncrementVarSublevelsUp_walker(Node *nod
phv->phlevelsup += context->delta_sublevels_up;
/* fall through to recurse into argument */
}
+ if (IsA(node, ReturningExpr))
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+
+ if (rexpr->retlevelsup >= context->min_sublevels_up)
+ rexpr->retlevelsup += context->delta_sublevels_up;
+ /* fall through to recurse into argument */
+ }
if (IsA(node, RangeTblEntry))
{
RangeTblEntry *rte = (RangeTblEntry *) node;
@@ -883,6 +891,68 @@ IncrementVarSublevelsUp_rtable(List *rta
QTW_EXAMINE_RTES_BEFORE);
}
+/*
+ * SetVarReturningType - adjust Var nodes for a specified varreturningtype.
+ *
+ * Find all Var nodes referring to the specified result relation in the given
+ * expression and set their varreturningtype to the specified value.
+ *
+ * NOTE: although this has the form of a walker, we cheat and modify the
+ * Var nodes in-place. The given expression tree should have been copied
+ * earlier to ensure that no unwanted side-effects occur!
+ */
+
+typedef struct
+{
+ int result_relation;
+ int sublevels_up;
+ VarReturningType returning_type;
+} SetVarReturningType_context;
+
+static bool
+SetVarReturningType_walker(Node *node, SetVarReturningType_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ Var *var = (Var *) node;
+
+ if (var->varno == context->result_relation &&
+ var->varlevelsup == context->sublevels_up)
+ var->varreturningtype = context->returning_type;
+
+ return false;
+ }
+
+ if (IsA(node, Query))
+ {
+ /* Recurse into subselects */
+ bool result;
+
+ context->sublevels_up++;
+ result = query_tree_walker((Query *) node, SetVarReturningType_walker,
+ (void *) context, 0);
+ context->sublevels_up--;
+ return result;
+ }
+ return expression_tree_walker(node, SetVarReturningType_walker,
+ (void *) context);
+}
+
+static void
+SetVarReturningType(Node *node, int result_relation, int sublevels_up,
+ VarReturningType returning_type)
+{
+ SetVarReturningType_context context;
+
+ context.result_relation = result_relation;
+ context.sublevels_up = sublevels_up;
+ context.returning_type = returning_type;
+
+ /* Expect to start with an expression */
+ SetVarReturningType_walker(node, &context);
+}
/*
* rangeTableEntry_used - detect whether an RTE is referenced somewhere
@@ -1653,6 +1723,15 @@ map_variable_attnos(Node *node,
* relation. This is needed to handle whole-row Vars referencing the target.
* We expand such Vars into RowExpr constructs.
*
+ * In addition, for INSERT/UPDATE/DELETE/MERGE queries, the caller must
+ * provide result_relation, the index of the result relation in the rewritten
+ * query. This is needed to handle OLD/NEW RETURNING list Vars referencing
+ * target_varno. When such Vars are expanded, their varreturningtype is
+ * copied onto any replacement Vars referencing result_relation. In addition,
+ * if the replacement expression from the targetlist is not simply a Var
+ * referencing result_relation, it is wrapped in a ReturningExpr node (causing
+ * the executor to return NULL if the OLD/NEW row doesn't exist).
+ *
* outer_hasSubLinks works the same as for replace_rte_variables().
*/
@@ -1660,6 +1739,7 @@ typedef struct
{
RangeTblEntry *target_rte;
List *targetlist;
+ int result_relation;
ReplaceVarsNoMatchOption nomatch_option;
int nomatch_varno;
} ReplaceVarsFromTargetList_context;
@@ -1684,10 +1764,13 @@ ReplaceVarsFromTargetList_callback(Var *
* dropped columns. If the var is RECORD (ie, this is a JOIN), then
* omit dropped columns. In the latter case, attach column names to
* the RowExpr for use of the executor and ruleutils.c.
+ *
+ * The varreturningtype is copied onto each individual field Var, so
+ * that it is handled correctly when we recurse.
*/
expandRTE(rcon->target_rte,
- var->varno, var->varlevelsup, var->location,
- (var->vartype != RECORDOID),
+ var->varno, var->varlevelsup, var->varreturningtype,
+ var->location, (var->vartype != RECORDOID),
&colnames, &fields);
/* Adjust the generated per-field Vars... */
fields = (List *) replace_rte_variables_mutator((Node *) fields,
@@ -1699,6 +1782,18 @@ ReplaceVarsFromTargetList_callback(Var *
rowexpr->colnames = (var->vartype == RECORDOID) ? colnames : NIL;
rowexpr->location = var->location;
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = var->varreturningtype == VAR_RETURNING_OLD;
+ rexpr->retexpr = (Expr *) rowexpr;
+
+ return (Node *) rexpr;
+ }
+
return (Node *) rowexpr;
}
@@ -1764,6 +1859,34 @@ ReplaceVarsFromTargetList_callback(Var *
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("NEW variables in ON UPDATE rules cannot reference columns that are part of a multiple assignment in the subject UPDATE command")));
+ /* Handle any OLD/NEW RETURNING list Vars */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ /*
+ * Copy varreturningtype onto any Vars in the tlist item that
+ * refer to result_relation (which had better be non-zero).
+ */
+ if (rcon->result_relation == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+
+ SetVarReturningType((Node *) newnode, rcon->result_relation,
+ var->varlevelsup, var->varreturningtype);
+
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (!IsA(newnode, Var) ||
+ ((Var *) newnode)->varno != rcon->result_relation ||
+ ((Var *) newnode)->varlevelsup != var->varlevelsup)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = (var->varreturningtype == VAR_RETURNING_OLD);
+ rexpr->retexpr = newnode;
+
+ newnode = (Expr *) rexpr;
+ }
+ }
+
return (Node *) newnode;
}
}
@@ -1773,6 +1896,7 @@ ReplaceVarsFromTargetList(Node *node,
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks)
@@ -1781,6 +1905,7 @@ ReplaceVarsFromTargetList(Node *node,
context.target_rte = target_rte;
context.targetlist = targetlist;
+ context.result_relation = result_relation;
context.nomatch_option = nomatch_option;
context.nomatch_varno = nomatch_varno;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
new file mode 100644
index 2177d17..bbef920
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -167,6 +167,8 @@ typedef struct
List *subplans; /* List of Plan trees for SubPlans */
List *ctes; /* List of CommonTableExpr nodes */
AppendRelInfo **appendrels; /* Array of AppendRelInfo nodes, or NULL */
+ char *returningOld; /* alias for OLD in RETURNING list */
+ char *returningNew; /* alias for NEW in RETURNING list */
/* Workspace for column alias assignment: */
bool unique_using; /* Are we making USING names globally unique */
List *using_names; /* List of assigned names for USING columns */
@@ -426,6 +428,7 @@ static void get_merge_query_def(Query *q
static void get_utility_query_def(Query *query, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context);
static void get_target_list(List *targetList, deparse_context *context);
+static void get_returning_clause(Query *query, deparse_context *context);
static void get_setop_query(Node *setOp, Query *query,
deparse_context *context);
static Node *get_rule_sortgroupclause(Index ref, List *tlist,
@@ -3779,6 +3782,10 @@ deparse_context_for_plan_tree(PlannedStm
* the most-closely-nested first. This is needed to resolve PARAM_EXEC
* Params. Note we assume that all the Plan nodes share the same rtable.
*
+ * For a ModifyTable plan, we might also need to resolve references to OLD/NEW
+ * variables in the RETURNING list, so we copy the alias names of the OLD and
+ * NEW rows from the ModifyTable plan node.
+ *
* Once this function has been called, deparse_expression() can be called on
* subsidiary expression(s) of the specified Plan node. To deparse
* expressions of a different Plan node in the same Plan tree, re-call this
@@ -3799,6 +3806,13 @@ set_deparse_context_plan(List *dpcontext
dpns->ancestors = ancestors;
set_deparse_plan(dpns, plan);
+ /* For ModifyTable, set aliases for OLD and NEW in RETURNING */
+ if (IsA(plan, ModifyTable))
+ {
+ dpns->returningOld = ((ModifyTable *) plan)->returningOld;
+ dpns->returningNew = ((ModifyTable *) plan)->returningNew;
+ }
+
return dpcontext;
}
@@ -3996,6 +4010,8 @@ set_deparse_for_query(deparse_namespace
dpns->subplans = NIL;
dpns->ctes = query->cteList;
dpns->appendrels = NULL;
+ dpns->returningOld = query->returningOld;
+ dpns->returningNew = query->returningNew;
/* Assign a unique relation alias to each RTE */
set_rtable_names(dpns, parent_namespaces, NULL);
@@ -4387,8 +4403,8 @@ set_relation_column_names(deparse_namesp
if (rte->rtekind == RTE_FUNCTION && rte->functions != NIL)
{
/* Since we're not creating Vars, rtindex etc. don't matter */
- expandRTE(rte, 1, 0, -1, true /* include dropped */ ,
- &colnames, NULL);
+ expandRTE(rte, 1, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , &colnames, NULL);
}
else
colnames = rte->eref->colnames;
@@ -6315,6 +6331,43 @@ get_target_list(List *targetList, depars
}
static void
+get_returning_clause(Query *query, deparse_context *context)
+{
+ StringInfo buf = context->buf;
+
+ if (query->returningList)
+ {
+ bool have_with = false;
+
+ appendContextKeyword(context, " RETURNING",
+ -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
+
+ /* Add WITH options, if they're not the defaults */
+ if (query->returningOld && strcmp(query->returningOld, "old") != 0)
+ {
+ appendStringInfo(buf, " WITH (OLD AS %s", query->returningOld);
+ have_with = true;
+ }
+ if (query->returningNew && strcmp(query->returningNew, "new") != 0)
+ {
+ if (have_with)
+ appendStringInfoString(buf, ", ");
+ else
+ {
+ appendStringInfoString(buf, " WITH (");
+ have_with = true;
+ }
+ appendStringInfo(buf, "NEW AS %s", query->returningNew);
+ }
+ if (have_with)
+ appendStringInfoChar(buf, ')');
+
+ /* Add the returning expressions themselves */
+ get_target_list(query->returningList, context);
+ }
+}
+
+static void
get_setop_query(Node *setOp, Query *query, deparse_context *context)
{
StringInfo buf = context->buf;
@@ -6988,11 +7041,7 @@ get_insert_query_def(Query *query, depar
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7044,11 +7093,7 @@ get_update_query_def(Query *query, depar
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7247,11 +7292,7 @@ get_delete_query_def(Query *query, depar
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7410,11 +7451,7 @@ get_merge_query_def(Query *query, depars
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7562,7 +7599,13 @@ get_variable(Var *var, int levelsup, boo
}
rte = rt_fetch(varno, dpns->rtable);
- refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+ if (var->varreturningtype == VAR_RETURNING_OLD)
+ refname = dpns->returningOld;
+ else if (var->varreturningtype == VAR_RETURNING_NEW)
+ refname = dpns->returningNew;
+ else
+ refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+
colinfo = deparse_columns_fetch(varno, dpns);
attnum = varattno;
}
@@ -7676,7 +7719,8 @@ get_variable(Var *var, int levelsup, boo
attname = get_rte_attribute_name(rte, attnum);
}
- need_prefix = (context->varprefix || attname == NULL);
+ need_prefix = (context->varprefix || attname == NULL ||
+ var->varreturningtype != VAR_RETURNING_DEFAULT);
/*
* If we're considering a plain Var in an ORDER BY (but not GROUP BY)
@@ -8727,6 +8771,7 @@ isSimpleNode(Node *node, Node *parentNod
case T_SQLValueFunction:
case T_XmlExpr:
case T_NextValueExpr:
+ case T_ReturningExpr:
case T_NullIfExpr:
case T_Aggref:
case T_GroupingFunc:
@@ -8849,6 +8894,7 @@ isSimpleNode(Node *node, Node *parentNod
case T_CoalesceExpr: /* own parentheses */
case T_MinMaxExpr: /* own parentheses */
case T_XmlExpr: /* own parentheses */
+ case T_ReturningExpr: /* own parentheses */
case T_NullIfExpr: /* other separators */
case T_Aggref: /* own parentheses */
case T_GroupingFunc: /* own parentheses */
@@ -8901,6 +8947,7 @@ isSimpleNode(Node *node, Node *parentNod
case T_CoalesceExpr: /* own parentheses */
case T_MinMaxExpr: /* own parentheses */
case T_XmlExpr: /* own parentheses */
+ case T_ReturningExpr: /* own parentheses */
case T_NullIfExpr: /* other separators */
case T_Aggref: /* own parentheses */
case T_GroupingFunc: /* own parentheses */
@@ -10258,6 +10305,17 @@ get_rule_expr(Node *node, deparse_contex
}
break;
+ case T_ReturningExpr:
+ /* Returns old/new.(expression) */
+ if (((ReturningExpr *) node)->retold)
+ appendStringInfoString(buf, "old.(");
+ else
+ appendStringInfoString(buf, "new.(");
+ get_rule_expr((Node *) ((ReturningExpr *) node)->retexpr,
+ context, showimplicit);
+ appendStringInfoChar(buf, ')');
+ break;
+
case T_PartitionBoundSpec:
{
PartitionBoundSpec *spec = (PartitionBoundSpec *) node;
diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h
new file mode 100644
index cd97dfa..e142958
--- a/src/include/executor/execExpr.h
+++ b/src/include/executor/execExpr.h
@@ -26,9 +26,9 @@ struct JsonConstructorExprState;
/* Bits in ExprState->flags (see also execnodes.h for public flag bits): */
/* expression's interpreter has been initialized */
-#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 1)
+#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 5)
/* jump-threading is in use */
-#define EEO_FLAG_DIRECT_THREADED (1 << 2)
+#define EEO_FLAG_DIRECT_THREADED (1 << 6)
/* Typical API for out-of-line evaluation subroutines */
typedef void (*ExecEvalSubroutine) (ExprState *state,
@@ -72,16 +72,22 @@ typedef enum ExprEvalOp
EEOP_INNER_FETCHSOME,
EEOP_OUTER_FETCHSOME,
EEOP_SCAN_FETCHSOME,
+ EEOP_OLD_FETCHSOME,
+ EEOP_NEW_FETCHSOME,
/* compute non-system Var value */
EEOP_INNER_VAR,
EEOP_OUTER_VAR,
EEOP_SCAN_VAR,
+ EEOP_OLD_VAR,
+ EEOP_NEW_VAR,
/* compute system Var value */
EEOP_INNER_SYSVAR,
EEOP_OUTER_SYSVAR,
EEOP_SCAN_SYSVAR,
+ EEOP_OLD_SYSVAR,
+ EEOP_NEW_SYSVAR,
/* compute wholerow Var */
EEOP_WHOLEROW,
@@ -94,6 +100,8 @@ typedef enum ExprEvalOp
EEOP_ASSIGN_INNER_VAR,
EEOP_ASSIGN_OUTER_VAR,
EEOP_ASSIGN_SCAN_VAR,
+ EEOP_ASSIGN_OLD_VAR,
+ EEOP_ASSIGN_NEW_VAR,
/* assign ExprState's resvalue/resnull to a column of its resultslot */
EEOP_ASSIGN_TMP,
@@ -178,6 +186,7 @@ typedef enum ExprEvalOp
EEOP_SQLVALUEFUNCTION,
EEOP_CURRENTOFEXPR,
EEOP_NEXTVALUEEXPR,
+ EEOP_RETURNINGEXPR,
EEOP_ARRAYEXPR,
EEOP_ARRAYCOERCE,
EEOP_ROW,
@@ -301,7 +310,7 @@ typedef struct ExprEvalStep
*/
union
{
- /* for EEOP_INNER/OUTER/SCAN_FETCHSOME */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_FETCHSOME */
struct
{
/* attribute number up to which to fetch (inclusive) */
@@ -314,13 +323,14 @@ typedef struct ExprEvalStep
const TupleTableSlotOps *kind;
} fetch;
- /* for EEOP_INNER/OUTER/SCAN_[SYS]VAR[_FIRST] */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_[SYS]VAR */
struct
{
/* attnum is attr number - 1 for regular VAR ... */
/* but it's just the normal (negative) attr number for SYSVAR */
int attnum;
Oid vartype; /* type OID of variable */
+ VarReturningType varreturningtype; /* return old/new/default */
} var;
/* for EEOP_WHOLEROW */
@@ -349,6 +359,13 @@ typedef struct ExprEvalStep
int resultnum;
} assign_tmp;
+ /* for EEOP_RETURNINGEXPR */
+ struct
+ {
+ uint8 nullflag; /* flag to test if OLD/NEW row is NULL */
+ int jumpdone; /* jump here if OLD/NEW row is NULL */
+ } returningexpr;
+
/* for EEOP_CONST */
struct
{
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
new file mode 100644
index 69c3ebf..ea1eed1
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -624,6 +624,7 @@ extern int ExecCleanTargetListLength(Lis
extern TupleTableSlot *ExecGetTriggerOldSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetTriggerNewSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetReturningSlot(EState *estate, ResultRelInfo *relInfo);
+extern TupleTableSlot *ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleConversionMap *ExecGetChildToRootMap(ResultRelInfo *resultRelInfo);
extern TupleConversionMap *ExecGetRootToChildMap(ResultRelInfo *resultRelInfo, EState *estate);
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
new file mode 100644
index b67d518..f36d61f
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -74,11 +74,20 @@ typedef Datum (*ExprStateEvalFunc) (stru
/* Bits in ExprState->flags (see also execExpr.h for private flag bits): */
/* expression is for use with ExecQual() */
#define EEO_FLAG_IS_QUAL (1 << 0)
+/* expression refers to OLD table columns */
+#define EEO_FLAG_HAS_OLD (1 << 1)
+/* expression refers to NEW table columns */
+#define EEO_FLAG_HAS_NEW (1 << 2)
+/* OLD table row is NULL in RETURNING list */
+#define EEO_FLAG_OLD_IS_NULL (1 << 3)
+/* NEW table row is NULL in RETURNING list */
+#define EEO_FLAG_NEW_IS_NULL (1 << 4)
typedef struct ExprState
{
NodeTag type;
+#define FIELDNO_EXPRSTATE_FLAGS 1
uint8 flags; /* bitmask of EEO_FLAG_* bits, see above */
/*
@@ -290,6 +299,12 @@ typedef struct ExprContext
#define FIELDNO_EXPRCONTEXT_DOMAINNULL 13
bool domainValue_isNull;
+ /* Tuples that OLD/NEW Var nodes in RETURNING may refer to */
+#define FIELDNO_EXPRCONTEXT_OLDTUPLE 14
+ TupleTableSlot *ecxt_oldtuple;
+#define FIELDNO_EXPRCONTEXT_NEWTUPLE 15
+ TupleTableSlot *ecxt_newtuple;
+
/* Link to containing EState (NULL if a standalone ExprContext) */
struct EState *ecxt_estate;
@@ -504,6 +519,7 @@ typedef struct ResultRelInfo
TupleTableSlot *ri_ReturningSlot; /* for trigger output tuples */
TupleTableSlot *ri_TrigOldSlot; /* for a trigger's old tuple */
TupleTableSlot *ri_TrigNewSlot; /* for a trigger's new tuple */
+ TupleTableSlot *ri_AllNullSlot; /* for RETURNING OLD/NEW */
/* FDW callback functions, if foreign table */
struct FdwRoutine *ri_FdwRoutine;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
new file mode 100644
index b40b661..68e6f7c
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -197,6 +197,16 @@ typedef struct Query
OnConflictExpr *onConflict; /* ON CONFLICT DO [NOTHING | UPDATE] */
+ /*
+ * The following three fields describe the contents of the RETURNING list
+ * for INSERT/UPDATE/DELETE/MERGE. If returningOld or returningNew are
+ * non-NULL, then returningList may contain entries referring to old/new
+ * values in the result relation; if they are NULL, the default old/new
+ * alias was masked by a user-supplied alias/table name, and returningList
+ * cannot return old/new values.
+ */
+ char *returningOld; /* alias for OLD in RETURNING list */
+ char *returningNew; /* alias for NEW in RETURNING list */
List *returningList; /* return-values list (of TargetEntry) */
List *groupClause; /* a list of SortGroupClause's */
@@ -1727,6 +1737,41 @@ typedef struct MergeWhenClause
} MergeWhenClause;
/*
+ * ReturningOptionKind -
+ * Possible kinds of option in RETURNING WITH(...) list
+ *
+ * Currently, this is used only for specifying OLD/NEW aliases.
+ */
+typedef enum ReturningOptionKind
+{
+ RETURNING_OPTION_OLD, /* specify alias for OLD in RETURNING */
+ RETURNING_OPTION_NEW, /* specify alias for NEW in RETURNING */
+} ReturningOptionKind;
+
+/*
+ * ReturningOption -
+ * An individual option in the RETURNING WITH(...) list
+ */
+typedef struct ReturningOption
+{
+ NodeTag type;
+ ReturningOptionKind option; /* specified option */
+ char *value; /* option's value */
+ ParseLoc location; /* token location, or -1 if unknown */
+} ReturningOption;
+
+/*
+ * ReturningClause -
+ * List of RETURNING expressions, together with any WITH(...) options
+ */
+typedef struct ReturningClause
+{
+ NodeTag type;
+ List *options; /* list of ReturningOption elements */
+ List *exprs; /* list of expressions to return */
+} ReturningClause;
+
+/*
* TriggerTransition -
* representation of transition row or table naming clause
*
@@ -2043,7 +2088,7 @@ typedef struct InsertStmt
List *cols; /* optional: names of the target columns */
Node *selectStmt; /* the source SELECT/VALUES, or NULL */
OnConflictClause *onConflictClause; /* ON CONFLICT clause */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
OverridingKind override; /* OVERRIDING clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
@@ -2060,7 +2105,7 @@ typedef struct DeleteStmt
RangeVar *relation; /* relation to delete from */
List *usingClause; /* optional using clause for more tables */
Node *whereClause; /* qualifications */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
@@ -2077,7 +2122,7 @@ typedef struct UpdateStmt
List *targetList; /* the target list (of ResTarget) */
Node *whereClause; /* qualifications */
List *fromClause; /* optional from clause for more tables */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
@@ -2094,7 +2139,7 @@ typedef struct MergeStmt
Node *sourceRelation; /* source relation */
Node *joinCondition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
new file mode 100644
index 52f29bc..015e260
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -238,6 +238,8 @@ typedef struct ModifyTable
List *resultRelations; /* integer list of RT indexes */
List *updateColnosLists; /* per-target-table update_colnos lists */
List *withCheckOptionLists; /* per-target-table WCO lists */
+ char *returningOld; /* alias for OLD in RETURNING lists */
+ char *returningNew; /* alias for NEW in RETURNING lists */
List *returningLists; /* per-target-table RETURNING tlists */
List *fdwPrivLists; /* per-target-table FDW private data lists */
Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
new file mode 100644
index b0ef195..567c555
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -223,6 +223,12 @@ typedef struct Expr
* Note that it affects the meaning of all of varno, varnullingrels, and
* varnosyn, all of which refer to the range table of that query level.
*
+ * varreturningtype is used for Vars that refer to the target relation in the
+ * RETURNING list of data-modifying queries. The default behavior is to
+ * return old values for DELETE operations and new values for INSERT and
+ * UPDATE operations, but it is also possible to explicitly request old/new
+ * values by referring to the target relation using the OLD/NEW aliases.
+ *
* In the parser, varnosyn and varattnosyn are either identical to
* varno/varattno, or they specify the column's position in an aliased JOIN
* RTE that hides the semantic referent RTE's refname. This is a syntactic
@@ -244,6 +250,14 @@ typedef struct Expr
#define PRS2_OLD_VARNO 1
#define PRS2_NEW_VARNO 2
+/* Returning behavior for Vars in RETURNING list */
+typedef enum VarReturningType
+{
+ VAR_RETURNING_DEFAULT, /* return OLD for DELETE, else return NEW */
+ VAR_RETURNING_OLD, /* return OLD for DELETE/UPDATE, else NULL */
+ VAR_RETURNING_NEW, /* return NEW for INSERT/UPDATE, else NULL */
+} VarReturningType;
+
typedef struct Var
{
Expr xpr;
@@ -279,6 +293,9 @@ typedef struct Var
*/
Index varlevelsup;
+ /* returning type of this var (see above) */
+ VarReturningType varreturningtype;
+
/*
* varnosyn/varattnosyn are ignored for equality, because Vars with
* different syntactic identifiers are semantically the same as long as
@@ -2128,6 +2145,30 @@ typedef struct InferenceElem
Oid inferopclass; /* OID of att opclass, or InvalidOid */
} InferenceElem;
+/*
+ * ReturningExpr - return OLD/NEW.(expression) in RETURNING list
+ *
+ * This is used when updating an auto-updatable view and returning a view
+ * column that is not simply a Var referring to the base relation. In such
+ * cases, OLD/NEW.viewcol can expand to an arbitrary expression, but the
+ * result is required to be NULL if the OLD/NEW row doesn't exist. To handle
+ * this, the rewriter wraps the expanded expression in a ReturningExpr, which
+ * is equivalent to "CASE WHEN (OLD/NEW row exists) THEN (expr) ELSE NULL".
+ *
+ * A similar situation can arise when rewriting the RETURNING clause of a
+ * rule, which may also contain arbitrary expressions.
+ *
+ * ReturningExpr nodes never appear in a parsed Query --- they are only ever
+ * inserted by the rewriter.
+ */
+typedef struct ReturningExpr
+{
+ Expr xpr;
+ int retlevelsup; /* > 0 if it belongs to outer query */
+ bool retold; /* true for OLD, false for NEW */
+ Expr *retexpr; /* expression to be returned */
+} ReturningExpr;
+
/*--------------------
* TargetEntry -
* a target entry (used in query target lists)
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
new file mode 100644
index 93e3dc7..a6ab887
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -198,6 +198,7 @@ extern void pull_varattnos(Node *node, I
extern List *pull_vars_of_level(Node *node, int levelsup);
extern bool contain_var_clause(Node *node);
extern bool contain_vars_of_level(Node *node, int levelsup);
+extern bool contain_vars_returning_old_or_new(Node *node);
extern int locate_var_of_level(Node *node, int levelsup);
extern List *pull_var_clause(Node *node, int flags);
extern Node *flatten_join_alias_vars(PlannerInfo *root, Query *query, Node *node);
diff --git a/src/include/optimizer/paramassign.h b/src/include/optimizer/paramassign.h
new file mode 100644
index 4026b74..89d2d07
--- a/src/include/optimizer/paramassign.h
+++ b/src/include/optimizer/paramassign.h
@@ -22,6 +22,8 @@ extern Param *replace_outer_agg(PlannerI
extern Param *replace_outer_grouping(PlannerInfo *root, GroupingFunc *grp);
extern Param *replace_outer_merge_support(PlannerInfo *root,
MergeSupportFunc *msf);
+extern Param *replace_outer_returning(PlannerInfo *root,
+ ReturningExpr *rexpr);
extern Param *replace_nestloop_param_var(PlannerInfo *root, Var *var);
extern Param *replace_nestloop_param_placeholdervar(PlannerInfo *root,
PlaceHolderVar *phv);
diff --git a/src/include/parser/analyze.h b/src/include/parser/analyze.h
new file mode 100644
index 28b66fc..37f3bd3
--- a/src/include/parser/analyze.h
+++ b/src/include/parser/analyze.h
@@ -44,8 +44,9 @@ extern List *transformInsertRow(ParseSta
bool strip_indirection);
extern List *transformUpdateTargetList(ParseState *pstate,
List *origTlist);
-extern List *transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind);
+extern void transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind);
extern Query *transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree);
extern Query *transformStmt(ParseState *pstate, Node *parseTree);
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
new file mode 100644
index 2375e95..ac1a7f5
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -295,6 +295,11 @@ struct ParseState
* of SQL:2008 requires us to do it this way. We also use p_lateral_ok to
* forbid LATERAL references to an UPDATE/DELETE target table.
*
+ * While processing the RETURNING clause, special namespace items are added to
+ * refer to the OLD and NEW state of the result relation. These namespace
+ * items have p_returning_type set appropriately, for use when creating Vars.
+ * For convenience, this information is duplicated on each namespace column.
+ *
* At no time should a namespace list contain two entries that conflict
* according to the rules in checkNameSpaceConflicts; but note that those
* are more complicated than "must have different alias names", so in practice
@@ -312,6 +317,7 @@ struct ParseNamespaceItem
bool p_cols_visible; /* Column names visible as unqualified refs? */
bool p_lateral_only; /* Is only visible to LATERAL expressions? */
bool p_lateral_ok; /* If so, does join type allow use? */
+ VarReturningType p_returning_type; /* Is OLD/NEW for use in RETURNING? */
};
/*
@@ -342,6 +348,7 @@ struct ParseNamespaceColumn
Oid p_vartype; /* pg_type OID */
int32 p_vartypmod; /* type modifier value */
Oid p_varcollid; /* OID of collation, or InvalidOid */
+ VarReturningType p_varreturningtype; /* for RETURNING OLD/NEW */
Index p_varnosyn; /* rangetable index of syntactic referent */
AttrNumber p_varattnosyn; /* attribute number of syntactic referent */
bool p_dontexpand; /* not included in star expansion */
diff --git a/src/include/parser/parse_relation.h b/src/include/parser/parse_relation.h
new file mode 100644
index 91fd8e2..3dcc1ab
--- a/src/include/parser/parse_relation.h
+++ b/src/include/parser/parse_relation.h
@@ -114,6 +114,7 @@ extern void errorMissingRTE(ParseState *
extern void errorMissingColumn(ParseState *pstate,
const char *relname, const char *colname, int location) pg_attribute_noreturn();
extern void expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
extern List *expandNSItemVars(ParseState *pstate, ParseNamespaceItem *nsitem,
diff --git a/src/include/rewrite/rewriteManip.h b/src/include/rewrite/rewriteManip.h
new file mode 100644
index ac6d204..15839ac
--- a/src/include/rewrite/rewriteManip.h
+++ b/src/include/rewrite/rewriteManip.h
@@ -89,6 +89,7 @@ extern Node *ReplaceVarsFromTargetList(N
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks);
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
new file mode 100644
index 86943ae..b5431e1
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -105,8 +105,8 @@ my %replace_line = (
'SHOW TRANSACTION ISOLATION LEVEL ecpg_into',
'VariableShowStmt SHOW SESSION AUTHORIZATION' =>
'SHOW SESSION AUTHORIZATION ecpg_into',
- 'returning_clause RETURNING target_list' =>
- 'RETURNING target_list opt_ecpg_into',
+ 'returning_clause RETURNING returning_with_clause target_list' =>
+ 'RETURNING returning_with_clause target_list opt_ecpg_into',
'ExecuteStmt EXECUTE name execute_param_clause' =>
'EXECUTE prepared_name execute_param_clause execute_rest',
'ExecuteStmt CREATE OptTemp TABLE create_as_target AS EXECUTE name execute_param_clause opt_with_data'
diff --git a/src/test/isolation/expected/merge-update.out b/src/test/isolation/expected/merge-update.out
new file mode 100644
index 3063c0c..677263d
--- a/src/test/isolation/expected/merge-update.out
+++ b/src/test/isolation/expected/merge-update.out
@@ -40,12 +40,12 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -98,14 +98,14 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -137,13 +137,13 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step a1: ABORT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+-------------------------
-UPDATE | 2|setup1 updated by merge2a
+merge_action|old |new |key|val
+------------+----------+-------------------------------+---+-------------------------
+UPDATE |(1,setup1)|(2,"setup1 updated by merge2a")| 2|setup1 updated by merge2a
(1 row)
step select2: SELECT * FROM target;
@@ -234,14 +234,14 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
-merge_action|key|val
-------------+---+--------------------------------------------------
-UPDATE | 2|initial updated by pa_merge1 updated by pa_merge2a
-UPDATE | 3|initial source not matched by pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+--------------------------------------------------------+---+--------------------------------------------------
+UPDATE |(1,"initial updated by pa_merge1")|(2,"initial updated by pa_merge1 updated by pa_merge2a")| 2|initial updated by pa_merge1 updated by pa_merge2a
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
(2 rows)
step pa_select2: SELECT * FROM pa_target;
@@ -273,7 +273,7 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
@@ -303,13 +303,13 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+-------------------------------------------------------------
-UPDATE | 3|initial source not matched by pa_merge2a
-UPDATE | 3|initial updated by pa_merge2 source not matched by pa_merge2a
-INSERT | 1|pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+-------------------------------------------------------------------+---+-------------------------------------------------------------
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
+UPDATE |(2,"initial updated by pa_merge2")|(3,"initial updated by pa_merge2 source not matched by pa_merge2a")| 3|initial updated by pa_merge2 source not matched by pa_merge2a
+INSERT | |(1,pa_merge2a) | 1|pa_merge2a
(3 rows)
step pa_select2: SELECT * FROM pa_target;
diff --git a/src/test/isolation/specs/merge-update.spec b/src/test/isolation/specs/merge-update.spec
new file mode 100644
index a33dcdb..c718ff6
--- a/src/test/isolation/specs/merge-update.spec
+++ b/src/test/isolation/specs/merge-update.spec
@@ -95,7 +95,7 @@ step "merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
step "merge2b"
{
@@ -128,7 +128,7 @@ step "pa_merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
# MERGE proceeds only if 'val' unchanged
step "pa_merge2b_when"
diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out
new file mode 100644
index 521d70a..57409ea
--- a/src/test/regress/expected/merge.out
+++ b/src/test/regress/expected/merge.out
@@ -297,13 +297,13 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
- merge_action | tid | balance
---------------+-----+---------
- DELETE | 1 | 10
- DELETE | 2 | 20
- DELETE | 3 | 30
- INSERT | 4 | 40
+RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ DELETE | (1,10) | | 1 | 10
+ DELETE | (2,20) | | 2 | 20
+ DELETE | (3,30) | | 3 | 30
+ INSERT | | (4,40) | 4 | 40
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -994,7 +994,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 T
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
NOTICE: BEFORE INSERT STATEMENT trigger
NOTICE: BEFORE UPDATE STATEMENT trigger
NOTICE: BEFORE DELETE STATEMENT trigger
@@ -1009,12 +1009,12 @@ NOTICE: AFTER UPDATE ROW trigger row: (
NOTICE: AFTER DELETE STATEMENT trigger
NOTICE: AFTER UPDATE STATEMENT trigger
NOTICE: AFTER INSERT STATEMENT trigger
- merge_action | tid | balance
---------------+-----+---------
- UPDATE | 3 | 10
- INSERT | 4 | 40
- DELETE | 2 | 20
- UPDATE | 1 | 0
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ UPDATE | (3,30) | (3,10) | 3 | 10
+ INSERT | | (4,40) | 4 | 40
+ DELETE | (2,20) | | 2 | 20
+ UPDATE | (1,10) | (1,0) | 1 | 0
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -1436,17 +1436,19 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
WHEN 'DELETE' THEN 'Removed '||t
END AS description;
- action | tid | balance | description
---------+-----+---------+---------------------
- del | 1 | 100 | Removed (1,100)
- upd | 2 | 220 | Added 20 to balance
- ins | 4 | 40 | Inserted (4,40)
+ action | old_tid | old_balance | new_tid | new_balance | delta_balance | tid | balance | description
+--------+---------+-------------+---------+-------------+---------------+-----+---------+---------------------
+ del | 1 | 100 | | | | 1 | 100 | Removed (1,100)
+ upd | 2 | 200 | 2 | 220 | 20 | 2 | 220 | Added 20 to balance
+ ins | | | 4 | 40 | | 4 | 40 | Inserted (4,40)
(3 rows)
ROLLBACK;
@@ -1473,7 +1475,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -1487,14 +1489,14 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
- action | log_action | tid | last_change
---------+------------+-----+---------------------
- DELETE | UPDATE | 1 | Removed (1,100)
- UPDATE | INSERT | 2 | Added 20 to balance
- INSERT | INSERT | 4 | Inserted (4,40)
+ action | old_data | new_data | tid | balance | description | log_action | old_log | new_log | tid | last_change
+--------+----------+----------+-----+---------+---------------------+------------+----------------------+---------------------------+-----+---------------------
+ DELETE | (1,100) | | 1 | 100 | Removed (1,100) | UPDATE | (1,"Original value") | (1,"Removed (1,100)") | 1 | Removed (1,100)
+ UPDATE | (2,200) | (2,220) | 2 | 220 | Added 20 to balance | INSERT | | (2,"Added 20 to balance") | 2 | Added 20 to balance
+ INSERT | | (4,40) | 4 | 40 | Inserted (4,40) | INSERT | | (4,"Inserted (4,40)") | 4 | Inserted (4,40)
(3 rows)
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -1518,11 +1520,11 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
-DELETE 1 100
-UPDATE 2 220
-INSERT 4 40
+DELETE 1 100 \N \N
+UPDATE 2 200 2 220
+INSERT \N \N 4 40
ROLLBACK;
-- SQL function with MERGE ... RETURNING
BEGIN;
@@ -2039,10 +2041,10 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
- merge_action | tid | balance | val
---------------+-----+---------+--------------------------
- UPDATE | 2 | 110 | initial updated by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance | val
+--------------+-----------------+------------------------------------+-----+---------+--------------------------
+ UPDATE | (1,100,initial) | (2,110,"initial updated by merge") | 2 | 110 | initial updated by merge
(1 row)
SELECT * FROM pa_target ORDER BY tid;
@@ -2324,18 +2326,18 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
- merge_action | logts | tid | balance | val
---------------+--------------------------+-----+---------+--------------------------
- UPDATE | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | logts | tid | balance | val
+--------------+--------------------------------------------+---------------------------------------------------------------+--------------------------+-----+---------+--------------------------
+ UPDATE | ("Tue Jan 31 00:00:00 2017",1,100,initial) | ("Tue Jan 31 00:00:00 2017",1,110,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",2,200,initial) | ("Tue Feb 28 00:00:00 2017",2,220,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",3,30,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",4,400,initial) | ("Tue Jan 31 00:00:00 2017",4,440,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",5,500,initial) | ("Tue Feb 28 00:00:00 2017",5,550,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",6,60,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",7,700,initial) | ("Tue Jan 31 00:00:00 2017",7,770,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",8,800,initial) | ("Tue Feb 28 00:00:00 2017",8,880,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",9,90,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
(9 rows)
SELECT * FROM pa_target ORDER BY tid;
diff --git a/src/test/regress/expected/returning.out b/src/test/regress/expected/returning.out
new file mode 100644
index cb51bb8..b4888db
--- a/src/test/regress/expected/returning.out
+++ b/src/test/regress/expected/returning.out
@@ -355,3 +355,511 @@ INSERT INTO foo AS bar DEFAULT VALUES RE
42
(1 row)
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+ERROR: syntax error at or near "nonsuch"
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS so...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+ERROR: table name "foo" specified more than once
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+ERROR: OLD cannot be specified multiple times
+LINE 1: ...EFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) ...
+ ^
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ -> Result
+ Output: 4, NULL::text, 42, '99'::bigint
+(4 rows)
+
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+------+----+----+----+----+----------+-------+----+----+----+----+----+----+----+----
+ | | | | | | foo | (0,4) | 4 | | 42 | 99 | 4 | | 42 | 99
+(1 row)
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (o.tableoid)::regclass, o.ctid, o.f1, o.f2, o.f3, o.f4, (n.tableoid)::regclass, n.ctid, n.f1, n.f2, n.f3, n.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ Conflict Resolution: UPDATE
+ Conflict Arbiter Indexes: foo_f1_idx
+ -> Values Scan on "*VALUES*"
+ Output: "*VALUES*".column1, "*VALUES*".column2, 42, '99'::bigint
+(6 rows)
+
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+----+----------+-------+----+------------+----+----+----+------------+----+----
+ foo | (0,4) | 4 | | 42 | 99 | foo | (0,5) | 4 | conflicted | -1 | 99 | 4 | conflicted | -1 | 99
+ | | | | | | foo | (0,6) | 5 | ok | 42 | 99 | 5 | ok | 42 | 99
+(2 rows)
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ QUERY PLAN
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, old.*, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, new.*, (((old.f4)::text || '->'::text) || (new.f4)::text)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(8 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ tableoid | ctid | f1 | f2 | f3 | f4 | old | tableoid | ctid | f1 | f2 | f3 | f4 | new | change
+----------+-------+----+----+----+----+--------------+----------+-------+----+----+----+-----+---------------+---------
+ foo | (0,6) | 5 | ok | 42 | 99 | (5,ok,42,99) | foo | (0,7) | 5 | ok | 42 | 100 | (5,ok,42,100) | 99->100
+(1 row)
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(6 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+-----+----------+------+----+----+----+----+----+----+----+-----
+ foo | (0,7) | 5 | ok | 42 | 100 | | | | | | | 5 | ok | 42 | 100
+(1 row)
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ -> Result
+ Output: 5, 'subquery test'::text, 42, '99'::bigint
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(16 rows)
+
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ | 109
+(1 row)
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2), (SubPlan 3)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Result
+ Output: (old.f4 = new.f4)
+ SubPlan 2
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 3
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(23 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ ?column? | old_max | new_max
+----------+---------+---------
+ f | 109 | 110
+(1 row)
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(18 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ 110 |
+(1 row)
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, new.f1, new.f2, new.f3, new.f4, foo_2.f1, foo_2.f2, foo_2.f3, foo_2.f4
+ Update on pg_temp.foo foo_2
+ -> Nested Loop
+ Output: (foo_2.f2 || ' (deleted)'::text), '-1'::integer, '-1'::bigint, foo_1.ctid, foo_1.tableoid, foo_2.tableoid, foo_2.ctid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f2, foo_2.f1, foo_2.tableoid, foo_2.ctid
+ Filter: (foo_2.f1 = 4)
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.ctid, foo_1.f1, foo_1.tableoid
+ Filter: (foo_1.f1 = 4)
+(11 rows)
+
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----+------------+----+----+----+----------------------+----+----+----+----------------------+----+----
+ 4 | conflicted | -1 | 99 | 4 | conflicted (deleted) | -1 | -1 | 4 | conflicted (deleted) | -1 | -1
+(1 row)
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, old.(joinme.other), new.f1, new.f2, new.f3, new.f4, new.(joinme.other), foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4, joinme.other, (new.f3 - old.f3)
+ Update on pg_temp.foo foo_1
+ -> Hash Join
+ Output: foo_2.f1, (foo_2.f3 + 1), joinme.ctid, foo_2.ctid, joinme_1.ctid, joinme.other, foo_1.tableoid, foo_1.ctid, foo_2.tableoid
+ Hash Cond: (foo_1.f2 = joinme.f2j)
+ -> Hash Join
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme_1.ctid, joinme_1.f2j
+ Hash Cond: (joinme_1.f2j = foo_1.f2)
+ -> Seq Scan on pg_temp.joinme joinme_1
+ Output: joinme_1.ctid, joinme_1.f2j
+ -> Hash
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Hash
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Hash Join
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Hash Cond: (joinme.f2j = foo_2.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.ctid, joinme.other, joinme.f2j
+ -> Hash
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Filter: (foo_2.f3 = 57)
+(27 rows)
+
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 57 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 1
+(1 row)
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ERROR: column reference "new.f1" is ambiguous
+LINE 3: RETURNING new.f1, new.f4
+ ^
+DETAIL: It could refer to either a PL/pgSQL variable or a table column.
+QUERY: UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4
+CONTEXT: PL/pgSQL function joinview_upd_trig_fn() line 4 at SQL statement
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.joinview
+ Output: old.f1, old.f2, old.f3, old.f4, old.other, new.f1, new.f2, new.f3, new.f4, new.other, joinview.f1, joinview.f2, joinview.f3, joinview.f4, joinview.other, (new.f3 - old.f3)
+ -> Hash Join
+ Output: (foo.f3 + 1), '7'::bigint, ROW(foo.f1, foo.f2, foo.f3, foo.f4, joinme.other), foo.ctid, joinme.ctid, foo.tableoid
+ Hash Cond: (joinme.f2j = foo.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.other, joinme.ctid, joinme.f2j
+ -> Hash
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ -> Seq Scan on pg_temp.foo
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ Filter: (foo.f3 = 58)
+(12 rows)
+
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 1
+(1 row)
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+NOTICE: drop cascades to 3 other objects
+DETAIL: drop cascades to rule voo_i on view voo
+drop cascades to view joinview
+drop cascades to rule foo_del_rule on table foo
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+ERROR: column old.f3 does not exist
+LINE 1: UPDATE foo SET f4 = f4 + 1 RETURNING old.f3;
+ ^
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+ old | new
+-------------------------------+------------------------------
+ (1,xxx,20) | (1,xxx,21)
+ (2,more,141) | (2,more,142)
+ (4,"conflicted (deleted)",-1) | (4,"conflicted (deleted)",0)
+ (3,zoo2,70) | (3,zoo2,71)
+(4 rows)
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+------+----------+-------+-------
+ | | zerocol | (0,1) | (0,1)
+(1 row)
+
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+-------+----------+------+-------
+ zerocol | (0,1) | | | (0,1)
+(1 row)
+
+DROP TABLE zerocol;
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+----------+------+---+---+---+-------------+-------+---+------+----+---+------+----
+ | | | | | foo_part_s1 | (0,1) | 1 | 17.1 | P1 | 1 | 17.1 | P1
+ | | | | | foo_part_s2 | (0,1) | 2 | 17.2 | P2 | 2 | 17.2 | P2
+ | | | | | foo_part_d1 | (0,1) | 3 | 17.3 | P3 | 3 | 17.3 | P3
+ | | | | | foo_part_d2 | (0,1) | 4 | 17.4 | P4 | 4 | 17.4 | P4
+(4 rows)
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_s1 | (0,1) | 1 | 17.1 | P1 | foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | 2 | 18.1 | P1->P2
+(1 row)
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_d1 | (0,1) | 3 | 17.3 | P3 | foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | 1 | 18.3 | P3->P1
+(1 row)
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+--------+-------------+-------+---+------+------------+---+------+------------
+ foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | 3 | 19.3 | P3->P1->P3
+(1 row)
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+------------+-------------+-------+---+------+----------------+---+------+----------------
+ foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | 4 | 20.3 | P3->P1->P3->P4
+(1 row)
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----------------+----------+------+---+---+---+---+------+----------------
+ foo_part_s2 | (0,1) | 2 | 17.2 | P2 | | | | | | 2 | 17.2 | P2
+ foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | | | | | | 2 | 18.1 | P1->P2
+ foo_part_d2 | (0,1) | 4 | 17.4 | P4 | | | | | | 4 | 17.4 | P4
+ foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | | | | | | 4 | 20.3 | P3->P1->P3->P4
+(4 rows)
+
+DROP TABLE foo_parted;
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o)
+ o.*, new.*, o, new, o.f1 = new.f1, o = new,
+ (SELECT o.f2 = new.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = new.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = new);
+END;
+\sf foo_update
+CREATE OR REPLACE FUNCTION public.foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ UPDATE foo SET f1 = (foo.f1 + 1)
+ RETURNING WITH (OLD AS o) o.f1,
+ o.f2,
+ o.f4,
+ new.f1,
+ new.f2,
+ new.f4,
+ o.*::foo AS o,
+ new.*::foo AS new,
+ (o.f1 = new.f1),
+ (o.* = new.*),
+ ( SELECT (o.f2 = new.f2)),
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f1 = o.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f4 = new.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = o.*)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = new.*)) AS count;
+END
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
new file mode 100644
index 2b47013..c4ebd67
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3645,7 +3645,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1
-- test deparsing
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3684,11 +3687,12 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
CREATE OR REPLACE FUNCTION public.merge_sf_test()
- RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[])
+ RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[], old_id integer, old_data text, old_filling integer[], new_id integer, new_data text, new_filling integer[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3726,12 +3730,18 @@ BEGIN ATOMIC
WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
- RETURNING MERGE_ACTION() AS action,
+ RETURNING WITH (OLD AS o, NEW AS n) MERGE_ACTION() AS action,
s.a,
s.b,
t.id,
t.data,
- t.filling;
+ t.filling,
+ o.id,
+ o.data,
+ o.filling,
+ n.id,
+ n.data,
+ n.filling;
END
CREATE FUNCTION merge_sf_test2()
RETURNS void
diff --git a/src/test/regress/expected/updatable_views.out b/src/test/regress/expected/updatable_views.out
new file mode 100644
index 8786058..bface0e
--- a/src/test/regress/expected/updatable_views.out
+++ b/src/test/regress/expected/updatable_views.out
@@ -437,7 +437,7 @@ NOTICE: drop cascades to view ro_view19
-- simple updatable view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS SELECT *, 'Const' AS c FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view1';
@@ -462,7 +462,8 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | YES
rw_view1 | b | YES
-(2 rows)
+ rw_view1 | c | NO
+(3 rows)
INSERT INTO rw_view1 VALUES (3, 'Row 3');
INSERT INTO rw_view1 (a) VALUES (4);
@@ -479,20 +480,22 @@ SELECT * FROM base_tbl;
5 | Unspecified
(6 rows)
+SET jit_above_cost = 0;
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+-------+---+-------------
- UPDATE | 1 | ROW 1 | 1 | ROW 1
- DELETE | 3 | ROW 3 | 3 | Row 3
- INSERT | 2 | ROW 2 | 2 | Unspecified
+ RETURNING merge_action(), v.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c
+--------------+---+-------+-------------------+-----------------------+---+-------------+-------
+ UPDATE | 1 | ROW 1 | (1,"Row 1",Const) | (1,"ROW 1",Const) | 1 | ROW 1 | Const
+ DELETE | 3 | ROW 3 | (3,"Row 3",Const) | | 3 | Row 3 | Const
+ INSERT | 2 | ROW 2 | | (2,Unspecified,Const) | 2 | Unspecified | Const
(3 rows)
+SET jit_above_cost TO DEFAULT;
SELECT * FROM base_tbl ORDER BY a;
a | b
----+-------------
@@ -511,13 +514,13 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------------
- UPDATE | 1 | R1 | 1 | R1
- DELETE | | | 5 | Unspecified
- DELETE | 2 | R2 | 2 | Unspecified
- INSERT | 3 | R3 | 3 | Unspecified
+ RETURNING merge_action(), v.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c
+--------------+---+----+-----------------------+-----------------------+---+-------------+-------
+ UPDATE | 1 | R1 | (1,"ROW 1",Const) | (1,R1,Const) | 1 | R1 | Const
+ DELETE | | | (5,Unspecified,Const) | | 5 | Unspecified | Const
+ DELETE | 2 | R2 | (2,Unspecified,Const) | | 2 | Unspecified | Const
+ INSERT | 3 | R3 | | (3,Unspecified,Const) | 3 | Unspecified | Const
(4 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -634,8 +637,10 @@ DROP TABLE base_tbl_hist;
-- view on top of view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view2';
@@ -660,27 +665,29 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view2 | aaa | YES
rw_view2 | bbb | YES
-(2 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(4 rows)
INSERT INTO rw_view2 VALUES (3, 'Row 3');
INSERT INTO rw_view2 (aaa) VALUES (4);
SELECT * FROM rw_view2;
- aaa | bbb
------+-------------
- 1 | Row 1
- 2 | Row 2
- 3 | Row 3
- 4 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Unspecified | Const1 | Const2
(4 rows)
UPDATE rw_view2 SET bbb='Row 4' WHERE aaa=4;
DELETE FROM rw_view2 WHERE aaa=2;
SELECT * FROM rw_view2;
- aaa | bbb
------+-------
- 1 | Row 1
- 3 | Row 3
- 4 | Row 4
+ aaa | bbb | c1 | c2
+-----+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Row 4 | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -688,20 +695,20 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-------------
- DELETE | 3 | R3 | 3 | Row 3
- UPDATE | 4 | R4 | 4 | R4
- INSERT | 5 | R5 | 5 | Unspecified
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+---------------------------+-------------------------------+-----+-------------+--------+--------
+ DELETE | 3 | R3 | (3,"Row 3",Const1,Const2) | | 3 | Row 3 | Const1 | Const2
+ UPDATE | 4 | R4 | (4,"Row 4",Const1,Const2) | (4,R4,Const1,Const2) | 4 | R4 | Const1 | Const2
+ INSERT | 5 | R5 | | (5,Unspecified,Const1,Const2) | 5 | Unspecified | Const1 | Const2
(3 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-------------
- 1 | Row 1
- 4 | R4
- 5 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 4 | R4 | Const1 | Const2
+ 5 | Unspecified | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -710,21 +717,21 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-----------------------
- UPDATE | | | 1 | Not matched by source
- DELETE | 4 | r4 | 4 | R4
- UPDATE | 5 | r5 | 5 | r5
- INSERT | 6 | r6 | 6 | Unspecified
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+-------------------------------+-------------------------------------------+-----+-----------------------+--------+--------
+ UPDATE | | | (1,"Row 1",Const1,Const2) | (1,"Not matched by source",Const1,Const2) | 1 | Not matched by source | Const1 | Const2
+ DELETE | 4 | r4 | (4,R4,Const1,Const2) | | 4 | R4 | Const1 | Const2
+ UPDATE | 5 | r5 | (5,Unspecified,Const1,Const2) | (5,r5,Const1,Const2) | 5 | r5 | Const1 | Const2
+ INSERT | 6 | r6 | | (6,Unspecified,Const1,Const2) | 6 | Unspecified | Const1 | Const2
(4 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-----------------------
- 1 | Not matched by source
- 5 | r5
- 6 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-----------------------+--------+--------
+ 1 | Not matched by source | Const1 | Const2
+ 5 | r5 | Const1 | Const2
+ 6 | Unspecified | Const1 | Const2
(3 rows)
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -886,16 +893,25 @@ SELECT table_name, column_name, is_updat
rw_view2 | b | YES
(4 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | a | b
+---+---+---+-------
+ | | 3 | Row 3
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+ a | b | a | b
+---+----+---+----
+ 3 | R3 | 3 | R3
+(1 row)
+
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+----+---+-----------
+ 3 | R3 | 3 | Row three
(1 row)
SELECT * FROM rw_view2;
@@ -906,10 +922,10 @@ SELECT * FROM rw_view2;
3 | Row three
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+-----------+---+---
+ 3 | Row three | |
(1 row)
SELECT * FROM rw_view2;
@@ -960,8 +976,10 @@ drop cascades to view rw_view2
-- view on top of view with triggers
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name LIKE 'rw_view%'
@@ -992,9 +1010,12 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE FUNCTION rw_view1_trig_fn()
RETURNS trigger AS
@@ -1002,9 +1023,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -1045,9 +1068,12 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_upd_trig INSTEAD OF UPDATE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1081,9 +1107,12 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_del_trig INSTEAD OF DELETE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1117,41 +1146,44 @@ SELECT table_name, column_name, is_updat
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+---+----+----+---+-------+----------------+--------
+ | | | | 3 | Row 3 | Trigger Const1 | Const2
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-------+--------+--------+---+-----------+----------------+--------
+ 3 | Row 3 | Const1 | Const2 | 3 | Row three | Trigger Const1 | Const2
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-----------
- 1 | Row 1
- 2 | Row 2
- 3 | Row three
+ a | b | c1 | c2
+---+-----------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row three | Const1 | Const2
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-----------+--------+--------+---+---+----+----
+ 3 | Row three | Const1 | Const2 | | | |
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-------
- 1 | Row 1
- 2 | Row 2
+ a | b | c1 | c2
+---+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
(2 rows)
MERGE INTO rw_view2 t
@@ -1159,12 +1191,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------
- DELETE | 1 | R1 | 1 | Row 1
- UPDATE | 2 | R2 | 2 | R2
- INSERT | 3 | R3 | 3 | R3
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+---------------------------+--------------------------------+---+-------+----------------+--------
+ DELETE | 1 | R1 | (1,"Row 1",Const1,Const2) | | 1 | Row 1 | Const1 | Const2
+ UPDATE | 2 | R2 | (2,"Row 2",Const1,Const2) | (2,R2,"Trigger Const1",Const2) | 2 | R2 | Trigger Const1 | Const2
+ INSERT | 3 | R3 | | (3,R3,"Trigger Const1",Const2) | 3 | R3 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -1182,12 +1214,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-----------------------
- UPDATE | 2 | r2 | 2 | r2
- UPDATE | | | 3 | Not matched by source
- INSERT | 1 | r1 | 1 | r1
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+----------------------+-----------------------------------------------------+---+-----------------------+----------------+--------
+ UPDATE | 2 | r2 | (2,R2,Const1,Const2) | (2,r2,"Trigger Const1",Const2) | 2 | r2 | Trigger Const1 | Const2
+ UPDATE | | | (3,R3,Const1,Const2) | (3,"Not matched by source","Trigger Const1",Const2) | 3 | Not matched by source | Trigger Const1 | Const2
+ INSERT | 1 | r1 | | (1,r1,"Trigger Const1",Const2) | 1 | r1 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql
new file mode 100644
index 5ddcca8..5967a82
--- a/src/test/regress/sql/merge.sql
+++ b/src/test/regress/sql/merge.sql
@@ -235,7 +235,7 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -677,7 +677,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 T
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -930,7 +930,9 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -956,7 +958,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -970,7 +972,7 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -988,7 +990,7 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
ROLLBACK;
@@ -1265,7 +1267,7 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
@@ -1456,7 +1458,7 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
diff --git a/src/test/regress/sql/returning.sql b/src/test/regress/sql/returning.sql
new file mode 100644
index a460f82..29841a9
--- a/src/test/regress/sql/returning.sql
+++ b/src/test/regress/sql/returning.sql
@@ -160,3 +160,205 @@ INSERT INTO foo AS bar DEFAULT VALUES RE
INSERT INTO foo AS bar DEFAULT VALUES RETURNING foo.*; -- fails, wrong name
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.*; -- ok
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.f3; -- ok
+
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DROP TABLE zerocol;
+
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DROP TABLE foo_parted;
+
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o)
+ o.*, new.*, o, new, o.f1 = new.f1, o = new,
+ (SELECT o.f2 = new.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = new.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = new);
+END;
+
+\sf foo_update
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
new file mode 100644
index 4a5fa50..fdd3ff1
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1294,7 +1294,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -1333,7 +1336,8 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
diff --git a/src/test/regress/sql/updatable_views.sql b/src/test/regress/sql/updatable_views.sql
new file mode 100644
index 93b693a..1f8b0ff
--- a/src/test/regress/sql/updatable_views.sql
+++ b/src/test/regress/sql/updatable_views.sql
@@ -154,7 +154,7 @@ DROP SEQUENCE uv_seq CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS SELECT *, 'Const' AS c FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -175,13 +175,18 @@ UPDATE rw_view1 SET a=5 WHERE a=4;
DELETE FROM rw_view1 WHERE b='Row 2';
SELECT * FROM base_tbl;
+SET jit_above_cost = 0;
+
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, t.*;
+
+SET jit_above_cost TO DEFAULT;
+
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view1 t
@@ -191,7 +196,7 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view1 SET a=6 WHERE a=5;
@@ -240,8 +245,10 @@ DROP TABLE base_tbl_hist;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -268,7 +275,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
MERGE INTO rw_view2 t
@@ -277,7 +284,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -362,10 +369,14 @@ SELECT table_name, column_name, is_updat
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t USING (VALUES (3, 'Row 3')) AS v(a,b) ON t.a = v.a
@@ -381,8 +392,10 @@ DROP TABLE base_tbl CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -407,9 +420,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -479,10 +494,10 @@ SELECT table_name, column_name, is_updat
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t
@@ -490,7 +505,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view2 t
@@ -498,7 +513,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
new file mode 100644
index 171a7dd..f807ee0
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2466,6 +2466,10 @@ RestrictInfo
Result
ResultRelInfo
ResultState
+ReturningClause
+ReturningExpr
+ReturningOption
+ReturningOptionKind
ReturnSetInfo
ReturnStmt
RevmapContents
@@ -2614,6 +2618,7 @@ SetOperationStmt
SetQuantifier
SetToDefault
SetupWorkerPtrType
+SetVarReturningType_context
ShDependObjectInfo
SharedAggInfo
SharedBitmapHeapInstrumentation
@@ -3084,6 +3089,7 @@ VariableSpace
VariableStatData
VariableSubstituteHook
Variables
+VarReturningType
Vector32
Vector8
VersionedQuery
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2024-11-28 11:45 ` Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2024-11-28 11:45 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Tue, 29 Oct 2024 at 13:05, Dean Rasheed <[email protected]> wrote:
>
> Rebased version attached. No other changes.
>
In the light of 7f798aca1d5df290aafad41180baea0ae311b4ee, I guess I
should remove various (void *) casts that this patch adds, copied from
old code. I'll wait a while though, just in case the buildfarm objects
to that other patch.
Regards,
Dean
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2025-01-01 08:19 ` Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2025-01-01 08:19 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Thu, 28 Nov 2024 at 11:45, Dean Rasheed <[email protected]> wrote:
>
> In the light of 7f798aca1d5df290aafad41180baea0ae311b4ee, I guess I
> should remove various (void *) casts that this patch adds, copied from
> old code.
Attached is an updated patch with some additional tidying up, plus the
following changes:
I decided to get rid of the fast-path ExecJust* functions, which I had
added by more-or-less copy-pasting existing code, without thinking too
hard. In practice, compared to the cost of a DML query, it seems
unlikely that these optimisations would be worth it, and AFAICS only
ExecJustAssign{Old|New}Var could ever be hit, and then only for a
RETURNING list containing a single Var, which seems unlikely to be
common. Also, these optimisations weren't free, because they were
adding a small number of additional cycles to
ExecReadyInterpretedExpr().
In the Query struct, the 2 new fields specifying the names of the old
and new aliases in the RETURNING clause were called "returningOld" and
"returningNew", but those names didn't seem quite right, so I've
changed them to "returningOldAlias" and "returningNewAlias", which
seems a little more consistent and descriptive of what they are. In
addition, I had overlooked the fact that they should have been marked
as query_jumble_ignore because, like other aliases in a Query, the
choice of alias names doesn't materially affect the query.
Looking again at ruleutils.c, a ReturningExpr node can only occur when
running EXPLAIN, not when deparsing a query, so I've added a comment
to make that clear. Thinking more about what the result should look
like in EXPLAIN, I think the best thing to do is to just output the
referenced expression, ignoring the nulling effects of the
ReturningExpr node, making the result simpler and more intuitive.
Regards,
Dean
Attachments:
[text/x-patch] v22-0001-Add-OLD-NEW-support-to-RETURNING-in-DML-queries.patch (243.6K, ../../CAEZATCWsN0JgjcAtrGLitj=JywODb40qPx7eHMAD4xKBxX_Odw@mail.gmail.com/2-v22-0001-Add-OLD-NEW-support-to-RETURNING-in-DML-queries.patch)
download | inline diff:
From ed3e7bb6b12d7aa5ecd5128b59b8d853e14491f4 Mon Sep 17 00:00:00 2001
From: Dean Rasheed <[email protected]>
Date: Mon, 23 Dec 2024 14:15:15 +0000
Subject: [PATCH v22] Add OLD/NEW support to RETURNING in DML queries.
This allows the RETURNING list of INSERT/UPDATE/DELETE/MERGE queries
to explicitly return old and new values by using the special aliases
"old" and "new", which are automatically added to the query (if not
already defined) while parsing its RETURNING list, allowing things
like:
RETURNING old.colname, new.colname, ...
RETURNING old.*, new.*
Additionally, a new syntax is supported, allowing the names "old" and
"new" to be changed to user-supplied alias names, e.g.:
RETURNING WITH (OLD AS o, NEW AS n) o.colname, n.colname, ...
This is useful when the names "old" and "new" are already defined,
such as inside trigger functions, allowing backwards compatibility to
be maintained -- the interpretation of any existing queries that
happen to already refer to relations called "old" or "new", or use
those as aliases for other relations, is not changed.
For an INSERT, old values will generally be NULL, and for a DELETE,
new values will generally be NULL, but that may change for an INSERT
with an ON CONFLICT ... DO UPDATE clause, or if a query rewrite rule
changes the command type. Therefore, we put no restrictions on the use
of old and new in any DML queries.
Dean Rasheed, reviewed by Jian He and Jeff Davis.
Discussion: https://postgr.es/m/CAEZATCWx0J0-v=Qjc6gXzR=KtsdvAE7Ow=D=mu50AgOe+pvisQ@mail.gmail.com
---
.../postgres_fdw/expected/postgres_fdw.out | 124 +++-
contrib/postgres_fdw/sql/postgres_fdw.sql | 25 +-
doc/src/sgml/dml.sgml | 41 +-
doc/src/sgml/ref/delete.sgml | 40 +-
doc/src/sgml/ref/insert.sgml | 54 +-
doc/src/sgml/ref/merge.sgml | 35 +-
doc/src/sgml/ref/update.sgml | 38 +-
doc/src/sgml/rules.sgml | 17 +
src/backend/executor/execExpr.c | 149 ++++-
src/backend/executor/execExprInterp.c | 199 ++++++-
src/backend/executor/execMain.c | 1 +
src/backend/executor/execUtils.c | 28 +
src/backend/executor/nodeModifyTable.c | 223 +++++++-
src/backend/jit/llvm/llvmjit_expr.c | 119 +++-
src/backend/nodes/makefuncs.c | 12 +-
src/backend/nodes/nodeFuncs.c | 46 +-
src/backend/optimizer/path/allpaths.c | 1 +
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/setrefs.c | 15 +
src/backend/optimizer/plan/subselect.c | 45 +-
src/backend/optimizer/prep/prepjointree.c | 3 +-
src/backend/optimizer/util/appendinfo.c | 21 +-
src/backend/optimizer/util/clauses.c | 3 +
src/backend/optimizer/util/paramassign.c | 47 ++
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/optimizer/util/var.c | 44 ++
src/backend/parser/analyze.c | 150 ++++-
src/backend/parser/gram.y | 57 +-
src/backend/parser/parse_clause.c | 2 +
src/backend/parser/parse_expr.c | 18 +-
src/backend/parser/parse_merge.c | 4 +-
src/backend/parser/parse_relation.c | 33 +-
src/backend/parser/parse_target.c | 4 +-
src/backend/rewrite/rewriteHandler.c | 9 +
src/backend/rewrite/rewriteManip.c | 128 ++++-
src/backend/utils/adt/ruleutils.c | 108 +++-
src/include/executor/execExpr.h | 25 +-
src/include/executor/executor.h | 1 +
src/include/nodes/execnodes.h | 16 +
src/include/nodes/parsenodes.h | 52 +-
src/include/nodes/plannodes.h | 2 +
src/include/nodes/primnodes.h | 40 ++
src/include/optimizer/optimizer.h | 1 +
src/include/optimizer/paramassign.h | 2 +
src/include/parser/analyze.h | 5 +-
src/include/parser/parse_node.h | 7 +
src/include/parser/parse_relation.h | 1 +
src/include/rewrite/rewriteManip.h | 1 +
src/interfaces/ecpg/preproc/parse.pl | 4 +-
src/test/isolation/expected/merge-update.out | 52 +-
src/test/isolation/specs/merge-update.spec | 4 +-
src/test/regress/expected/merge.out | 96 ++--
src/test/regress/expected/returning.out | 540 ++++++++++++++++++
src/test/regress/expected/rules.out | 20 +-
src/test/regress/expected/updatable_views.out | 240 ++++----
src/test/regress/sql/merge.sql | 18 +-
src/test/regress/sql/returning.sql | 209 +++++++
src/test/regress/sql/rules.sql | 8 +-
src/test/regress/sql/updatable_views.sql | 50 +-
src/tools/pgindent/typedefs.list | 6 +
60 files changed, 2879 insertions(+), 388 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bf322198a2..687367896b 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -4975,12 +4975,12 @@ INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
- c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
-------+-----+-----+----+----+----+------------+----
- 1101 | 201 | aaa | | | | ft2 |
- 1102 | 202 | bbb | | | | ft2 |
- 1103 | 203 | ccc | | | | ft2 |
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
+ old | new | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+-----+---------------------------------+----+----+----+----+----+----+----+----+------+-----+-----+----+----+----+------------+----
+ | (1101,201,aaa,,,,"ft2 ",) | | | | | | | | | 1101 | 201 | aaa | | | | ft2 |
+ | (1102,202,bbb,,,,"ft2 ",) | | | | | | | | | 1102 | 202 | bbb | | | | ft2 |
+ | (1103,203,ccc,,,,"ft2 ",) | | | | | | | | | 1103 | 203 | ccc | | | | ft2 |
(3 rows)
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
@@ -5111,6 +5111,31 @@ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING
1017 | 507 | 0001700017_update7 | | | | ft2 |
(102 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.c1, old.c2, old.c3, old.c4, old.c5, old.c6, old.c7, old.c8, new.c1, new.c2, new.c3, new.c4, new.c5, new.c6, new.c7, new.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c2 = $2, c3 = $3 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: (c2 + 400), (c3 || '_update7b'::text), ctid, ft2.*
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 7)) FOR UPDATE
+(6 rows)
+
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+----+-----+---------------+------------------------------+--------------------------+----+------------+-----+----+-----+------------------------+------------------------------+--------------------------+----+------------+-----
+ 7 | 407 | 00007_update7 | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo | 7 | 807 | 00007_update7_update7b | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo
+ 17 | 407 | 00017_update7 | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo | 17 | 807 | 00017_update7_update7b | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo
+ 27 | 407 | 00027_update7 | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo | 27 | 807 | 00027_update7_update7b | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo
+ 37 | 407 | 00037_update7 | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo | 37 | 807 | 00037_update7_update7b | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -5241,6 +5266,29 @@ DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4;
1105 |
(103 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------
+ Delete on public.ft2
+ Output: old.c1, c4
+ Remote SQL: DELETE FROM "S 1"."T 1" WHERE ctid = $1 RETURNING "C 1", c4
+ -> Foreign Scan on public.ft2
+ Output: ctid
+ Remote SQL: SELECT ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 6)) FOR UPDATE
+(6 rows)
+
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ c1 | c4
+----+------------------------------
+ 6 | Wed Jan 07 00:00:00 1970 PST
+ 16 | Sat Jan 17 00:00:00 1970 PST
+ 26 | Tue Jan 27 00:00:00 1970 PST
+ 36 | Fri Feb 06 00:00:00 1970 PST
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
QUERY PLAN
@@ -6165,6 +6213,70 @@ UPDATE ft2 SET c3 = 'foo'
(1296,96,foo,,,,"ft2 ",) | 1296 | 96 | foo | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
(16 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.*, new.*, ft2.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan
+ Output: 'bar'::text, ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Relations: ((public.ft2) INNER JOIN (public.ft4)) INNER JOIN (public.ft5)
+ Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2.c1, r2.c2, r2.c3) END, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r2.c1, r2.c2, r2.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r2 ON (((r1.c2 = r2.c1)) AND ((r1."C 1" > 1200)))) INNER JOIN "S 1"."T 4" r3 ON (((r2.c1 = r3.c1)))) FOR UPDATE OF r1
+ -> Nested Loop
+ Output: ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Join Filter: (ft4.c1 = ft5.c1)
+ -> Sort
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Sort Key: ft2.c2
+ -> Hash Join
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Hash Cond: (ft2.c2 = ft4.c1)
+ -> Foreign Scan on public.ft2
+ Output: ft2.ctid, ft2.*, ft2.c2
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" > 1200)) FOR UPDATE
+ -> Hash
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ -> Foreign Scan on public.ft4
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 3"
+ -> Materialize
+ Output: ft5.*, ft5.c1
+ -> Foreign Scan on public.ft5
+ Output: ft5.*, ft5.c1
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 4"
+(29 rows)
+
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ old | new | ft2 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | ft4 | c1 | c2 | c3
+--------------------------------+--------------------------------+--------------------------------+------+----+-----+----+----+----+------------+----+----------------+----+----+--------
+ (1206,6,foo,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | 1206 | 6 | bar | | | | ft2 | | (6,7,AAA006) | 6 | 7 | AAA006
+ (1212,12,foo,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | 1212 | 12 | bar | | | | ft2 | | (12,13,AAA012) | 12 | 13 | AAA012
+ (1224,24,foo,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | 1224 | 24 | bar | | | | ft2 | | (24,25,AAA024) | 24 | 25 | AAA024
+ (1230,30,foo,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | 1230 | 30 | bar | | | | ft2 | | (30,31,AAA030) | 30 | 31 | AAA030
+ (1242,42,foo,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | 1242 | 42 | bar | | | | ft2 | | (42,43,AAA042) | 42 | 43 | AAA042
+ (1248,48,foo,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | 1248 | 48 | bar | | | | ft2 | | (48,49,AAA048) | 48 | 49 | AAA048
+ (1260,60,foo,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | 1260 | 60 | bar | | | | ft2 | | (60,61,AAA060) | 60 | 61 | AAA060
+ (1266,66,foo,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | 1266 | 66 | bar | | | | ft2 | | (66,67,AAA066) | 66 | 67 | AAA066
+ (1278,78,foo,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | 1278 | 78 | bar | | | | ft2 | | (78,79,AAA078) | 78 | 79 | AAA078
+ (1284,84,foo,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | 1284 | 84 | bar | | | | ft2 | | (84,85,AAA084) | 84 | 85 | AAA084
+ (1296,96,foo,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | 1296 | 96 | bar | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
+ (1218,18,foo,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | 1218 | 18 | bar | | | | ft2 | | (18,19,AAA018) | 18 | 19 | AAA018
+ (1236,36,foo,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | 1236 | 36 | bar | | | | ft2 | | (36,37,AAA036) | 36 | 37 | AAA036
+ (1254,54,foo,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | 1254 | 54 | bar | | | | ft2 | | (54,55,AAA054) | 54 | 55 | AAA054
+ (1272,72,foo,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | 1272 | 72 | bar | | | | ft2 | | (72,73,AAA072) | 72 | 73 | AAA072
+ (1290,90,foo,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | 1290 | 90 | bar | | | | ft2 | | (90,91,AAA090) | 90 | 91 | AAA090
+(16 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 3900522ccb..b58ab6ee58 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1469,7 +1469,7 @@ EXPLAIN (verbose, costs off)
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 300, c3 = c3 || '_update3' WHERE c1 % 10 = 3; -- can be pushed down
@@ -1477,6 +1477,13 @@ UPDATE ft2 SET c2 = c2 + 300, c3 = c3 || '_update3' WHERE c1 % 10 = 3;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *; -- can be pushed down
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -1485,6 +1492,11 @@ UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
EXPLAIN (verbose, costs off)
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4; -- can be pushed down
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2;
@@ -1511,6 +1523,17 @@ UPDATE ft2 SET c3 = 'foo'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
RETURNING ft2, ft2.*, ft4, ft4.*;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/doc/src/sgml/dml.sgml b/doc/src/sgml/dml.sgml
index 3d95bdb94e..458aee788b 100644
--- a/doc/src/sgml/dml.sgml
+++ b/doc/src/sgml/dml.sgml
@@ -308,7 +308,8 @@ DELETE FROM products;
</para>
<para>
- In an <command>INSERT</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>INSERT</command>, the default data available to
+ <literal>RETURNING</literal> is
the row as it was inserted. This is not so useful in trivial inserts,
since it would just repeat the data provided by the client. But it can
be very handy when relying on computed default values. For example,
@@ -325,7 +326,8 @@ INSERT INTO users (firstname, lastname) VALUES ('Joe', 'Cool') RETURNING id;
</para>
<para>
- In an <command>UPDATE</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>UPDATE</command>, the default data available to
+ <literal>RETURNING</literal> is
the new content of the modified row. For example:
<programlisting>
UPDATE products SET price = price * 1.10
@@ -335,7 +337,8 @@ UPDATE products SET price = price * 1.10
</para>
<para>
- In a <command>DELETE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>DELETE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the deleted row. For example:
<programlisting>
DELETE FROM products
@@ -345,7 +348,8 @@ DELETE FROM products
</para>
<para>
- In a <command>MERGE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>MERGE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the source row plus the content of the inserted, updated, or
deleted target row. Since it is quite common for the source and target to
have many of the same columns, specifying <literal>RETURNING *</literal>
@@ -359,6 +363,35 @@ MERGE INTO products p USING new_products n ON p.product_no = n.product_no
</programlisting>
</para>
+ <para>
+ In each of these commands, it is also possible to explicitly return the
+ old and new content of the modified row. For example:
+<programlisting>
+UPDATE products SET price = price * 1.10
+ WHERE price <= 99.99
+ RETURNING name, old.price AS old_price, new.price AS new_price,
+ new.price - old.price AS price_change;
+</programlisting>
+ In this example, writing <literal>new.price</literal> is the same as
+ just writing <literal>price</literal>, but it makes the meaning clearer.
+ </para>
+
+ <para>
+ This syntax for returning old and new values is available in
+ <command>INSERT</command>, <command>UPDATE</command>,
+ <command>DELETE</command>, and <command>MERGE</command> commands, but
+ typically old values will be <literal>NULL</literal> for an
+ <command>INSERT</command>, and new values will be <literal>NULL</literal>
+ for a <command>DELETE</command>. However, there are situations where it
+ can still be useful for those commands. For example, in an
+ <command>INSERT</command> with an
+ <link linkend="sql-on-conflict"><literal>ON CONFLICT DO UPDATE</literal></link>
+ clause, the old values will be non-<literal>NULL</literal> for conflicting
+ rows. Similarly, if a <command>DELETE</command> is turned into an
+ <command>UPDATE</command> by a <link linkend="sql-createrule">rewrite rule</link>,
+ the new values may be non-<literal>NULL</literal>.
+ </para>
+
<para>
If there are triggers (<xref linkend="triggers"/>) on the target table,
the data available to <literal>RETURNING</literal> is the row as modified by
diff --git a/doc/src/sgml/ref/delete.sgml b/doc/src/sgml/ref/delete.sgml
index 7717855bc9..29649f6afd 100644
--- a/doc/src/sgml/ref/delete.sgml
+++ b/doc/src/sgml/ref/delete.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
[ USING <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -160,6 +161,26 @@ DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ *
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -170,6 +191,23 @@ DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ *
or table(s) listed in <literal>USING</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return old values.
+ </para>
+
+ <para>
+ For a simple <command>DELETE</command>, all new values will be
+ <literal>NULL</literal>. However, if an <literal>ON DELETE</literal>
+ rule causes an <command>INSERT</command> or <command>UPDATE</command>
+ to be executed instead, the new values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml
index 6f0adee1a1..3f13991779 100644
--- a/doc/src/sgml/ref/insert.sgml
+++ b/doc/src/sgml/ref/insert.sgml
@@ -26,7 +26,8 @@ INSERT INTO <replaceable class="parameter">table_name</replaceable> [ AS <replac
[ OVERRIDING { SYSTEM | USER } VALUE ]
{ DEFAULT VALUES | VALUES ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="parameter">query</replaceable> }
[ ON CONFLICT [ <replaceable class="parameter">conflict_target</replaceable> ] <replaceable class="parameter">conflict_action</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">conflict_target</replaceable> can be one of:</phrase>
@@ -293,6 +294,26 @@ INSERT INTO <replaceable class="parameter">table_name</replaceable> [ AS <replac
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -305,6 +326,23 @@ INSERT INTO <replaceable class="parameter">table_name</replaceable> [ AS <replac
<literal>*</literal> to return all columns of the inserted or updated
row(s).
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values.
+ </para>
+
+ <para>
+ For a simple <command>INSERT</command>, all old values will be
+ <literal>NULL</literal>. However, for an <command>INSERT</command>
+ with an <literal>ON CONFLICT DO UPDATE</literal> clause, the old
+ values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
@@ -711,6 +749,20 @@ INSERT INTO employees_log SELECT *, current_timestamp FROM upd;
INSERT INTO distributors (did, dname)
VALUES (5, 'Gizmo Transglobal'), (6, 'Associated Computing, Inc')
ON CONFLICT (did) DO UPDATE SET dname = EXCLUDED.dname;
+</programlisting>
+ </para>
+ <para>
+ Insert or update new distributors as above, returning information
+ about any existing values that were updated, together with the new data
+ inserted. Note that the returned values for <literal>old_did</literal>
+ and <literal>old_dname</literal> will be <literal>NULL</literal> for
+ non-conflicting rows:
+<programlisting>
+INSERT INTO distributors (did, dname)
+ VALUES (5, 'Gizmo Transglobal'), (6, 'Associated Computing, Inc')
+ ON CONFLICT (did) DO UPDATE SET dname = EXCLUDED.dname
+ RETURNING old.did AS old_did, old.dname AS old_dname,
+ new.did AS new_did, new.dname AS new_dname;
</programlisting>
</para>
<para>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index d80a5c5cc9..3da0cd9e8e 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
MERGE INTO [ ONLY ] <replaceable class="parameter">target_table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">target_alias</replaceable> ]
USING <replaceable class="parameter">data_source</replaceable> ON <replaceable class="parameter">join_condition</replaceable>
<replaceable class="parameter">when_clause</replaceable> [...]
-[ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+[ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">data_source</replaceable> is:</phrase>
@@ -499,6 +500,25 @@ DELETE
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -517,6 +537,17 @@ DELETE
qualifying the <literal>*</literal> with the name or alias of the source
or target table.
</para>
+ <para>
+ A column name or <literal>*</literal> may also be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values from the target table to be returned. An unqualified column
+ name, or <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values
+ for <literal>INSERT</literal> and <literal>UPDATE</literal> actions, and
+ old values for <literal>DELETE</literal> actions.
+ </para>
</listitem>
</varlistentry>
@@ -722,7 +753,7 @@ WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN
UPDATE SET stock = w.stock + s.stock_delta
WHEN MATCHED THEN
DELETE
-RETURNING merge_action(), w.*;
+RETURNING merge_action(), w.winename, old.stock AS old_stock, new.stock AS new_stock;
</programlisting>
The <literal>wine_stock_changes</literal> table might be, for example, a
diff --git a/doc/src/sgml/ref/update.sgml b/doc/src/sgml/ref/update.sgml
index 1c433bec2b..12ec5ba070 100644
--- a/doc/src/sgml/ref/update.sgml
+++ b/doc/src/sgml/ref/update.sgml
@@ -29,7 +29,8 @@ UPDATE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [
} [, ...]
[ FROM <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -211,6 +212,26 @@ UPDATE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -221,6 +242,16 @@ UPDATE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [
or table(s) listed in <literal>FROM</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return new values.
+ </para>
</listitem>
</varlistentry>
@@ -348,12 +379,13 @@ UPDATE weather SET temp_lo = temp_lo+1, temp_hi = temp_lo+15, prcp = DEFAULT
</para>
<para>
- Perform the same operation and return the updated entries:
+ Perform the same operation and return the updated entries, and the old
+ precipitation value:
<programlisting>
UPDATE weather SET temp_lo = temp_lo+1, temp_hi = temp_lo+15, prcp = DEFAULT
WHERE city = 'San Francisco' AND date = '2003-07-03'
- RETURNING temp_lo, temp_hi, prcp;
+ RETURNING temp_lo, temp_hi, prcp, old.prcp AS old_prcp;
</programlisting>
</para>
diff --git a/doc/src/sgml/rules.sgml b/doc/src/sgml/rules.sgml
index 7a928bd7b9..e992baa91c 100644
--- a/doc/src/sgml/rules.sgml
+++ b/doc/src/sgml/rules.sgml
@@ -1645,6 +1645,23 @@ CREATE RULE shoelace_ins AS ON INSERT TO shoelace
<literal>RETURNING</literal> clause is simply ignored for <command>INSERT</command>.
</para>
+ <para>
+ Note that in the <literal>RETURNING</literal> clause of a rule,
+ <literal>OLD</literal> and <literal>NEW</literal> refer to the
+ pseudorelations added as extra range table entries to the rewritten
+ query, rather than old/new rows in the result relation. Thus, for
+ example, in a rule supporting <command>UPDATE</command> queries on this
+ view, if the <literal>RETURNING</literal> clause contained
+ <literal>old.sl_name</literal>, the old name would always be returned,
+ regardless of whether the <literal>RETURNING</literal> clause in the
+ query on the view specified <literal>OLD</literal> or <literal>NEW</literal>,
+ which might be confusing. To avoid this confusion, and support returning
+ old and new values in queries on the view, the <literal>RETURNING</literal>
+ clause in the rule definition should refer to entries from the result
+ relation such as <literal>shoelace_data.sl_name</literal>, without
+ specifying <literal>OLD</literal> or <literal>NEW</literal>.
+ </para>
+
<para>
Now assume that once in a while, a pack of shoelaces arrives at
the shop and a big parts list along with it. But you don't want
diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c
index 3d01a90bd6..65dc8beb66 100644
--- a/src/backend/executor/execExpr.c
+++ b/src/backend/executor/execExpr.c
@@ -55,10 +55,15 @@
typedef struct ExprSetupInfo
{
- /* Highest attribute numbers fetched from inner/outer/scan tuple slots: */
+ /*
+ * Highest attribute numbers fetched from inner/outer/scan/old/new tuple
+ * slots:
+ */
AttrNumber last_inner;
AttrNumber last_outer;
AttrNumber last_scan;
+ AttrNumber last_old;
+ AttrNumber last_new;
/* MULTIEXPR SubPlan nodes appearing in the expression: */
List *multiexpr_subplans;
} ExprSetupInfo;
@@ -446,8 +451,25 @@ ExecBuildProjectionInfo(List *targetList,
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+
+ /*
+ * Get the tuple from the relation being scanned, or the
+ * old/new tuple slot, if old/new values were requested.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+ break;
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_ASSIGN_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_ASSIGN_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ }
break;
}
@@ -535,7 +557,7 @@ ExecBuildUpdateProjection(List *targetList,
int nAssignableCols;
bool sawJunk;
Bitmapset *assignedCols;
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
ExprEvalStep scratch = {0};
int outerattnum;
ListCell *lc,
@@ -924,6 +946,7 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* system column */
scratch.d.var.attnum = variable->varattno;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -936,7 +959,20 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_SYSVAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ scratch.opcode = EEOP_SCAN_SYSVAR;
+ break;
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ }
break;
}
}
@@ -945,6 +981,7 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* regular user column */
scratch.d.var.attnum = variable->varattno - 1;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -957,7 +994,20 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_VAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ scratch.opcode = EEOP_SCAN_VAR;
+ break;
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ }
break;
}
}
@@ -2575,6 +2625,28 @@ ExecInitExprRec(Expr *node, ExprState *state,
break;
}
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ int retstep;
+
+ /* Skip expression evaluation if OLD/NEW row doesn't exist */
+ scratch.opcode = EEOP_RETURNINGEXPR;
+ scratch.d.returningexpr.nullflag = rexpr->retold ?
+ EEO_FLAG_OLD_IS_NULL : EEO_FLAG_NEW_IS_NULL;
+ scratch.d.returningexpr.jumpdone = -1; /* set below */
+ ExprEvalPushStep(state, &scratch);
+ retstep = state->steps_len - 1;
+
+ /* Steps to evaluate expression to return */
+ ExecInitExprRec(rexpr->retexpr, state, resv, resnull);
+
+ /* Jump target used if OLD/NEW row doesn't exist */
+ state->steps[retstep].d.returningexpr.jumpdone = state->steps_len;
+
+ break;
+ }
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(node));
@@ -2786,7 +2858,7 @@ ExecInitSubPlanExpr(SubPlan *subplan,
static void
ExecCreateExprSetupSteps(ExprState *state, Node *node)
{
- ExprSetupInfo info = {0, 0, 0, NIL};
+ ExprSetupInfo info = {0, 0, 0, 0, 0, NIL};
/* Prescan to find out what we need. */
expr_setup_walker(node, &info);
@@ -2809,8 +2881,8 @@ ExecPushExprSetupSteps(ExprState *state, ExprSetupInfo *info)
scratch.resnull = NULL;
/*
- * Add steps deforming the ExprState's inner/outer/scan slots as much as
- * required by any Vars appearing in the expression.
+ * Add steps deforming the ExprState's inner/outer/scan/old/new slots as
+ * much as required by any Vars appearing in the expression.
*/
if (info->last_inner > 0)
{
@@ -2842,6 +2914,26 @@ ExecPushExprSetupSteps(ExprState *state, ExprSetupInfo *info)
if (ExecComputeSlotInfo(state, &scratch))
ExprEvalPushStep(state, &scratch);
}
+ if (info->last_old > 0)
+ {
+ scratch.opcode = EEOP_OLD_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_old;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
+ if (info->last_new > 0)
+ {
+ scratch.opcode = EEOP_NEW_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_new;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
/*
* Add steps to execute any MULTIEXPR SubPlans appearing in the
@@ -2888,7 +2980,18 @@ expr_setup_walker(Node *node, ExprSetupInfo *info)
/* INDEX_VAR is handled by default case */
default:
- info->last_scan = Max(info->last_scan, attnum);
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ info->last_scan = Max(info->last_scan, attnum);
+ break;
+ case VAR_RETURNING_OLD:
+ info->last_old = Max(info->last_old, attnum);
+ break;
+ case VAR_RETURNING_NEW:
+ info->last_new = Max(info->last_new, attnum);
+ break;
+ }
break;
}
return false;
@@ -2926,6 +3029,11 @@ expr_setup_walker(Node *node, ExprSetupInfo *info)
* evaluation of the expression will have the same type of slot, with an
* equivalent descriptor.
*
+ * EEOP_OLD_FETCHSOME and EEOP_NEW_FETCHSOME are used to process RETURNING, if
+ * OLD/NEW columns are referred to explicitly. In both cases, the tuple
+ * descriptor comes from the parent scan node, so we treat them the same as
+ * EEOP_SCAN_FETCHSOME.
+ *
* Returns true if the deforming step is required, false otherwise.
*/
static bool
@@ -2939,7 +3047,9 @@ ExecComputeSlotInfo(ExprState *state, ExprEvalStep *op)
Assert(opcode == EEOP_INNER_FETCHSOME ||
opcode == EEOP_OUTER_FETCHSOME ||
- opcode == EEOP_SCAN_FETCHSOME);
+ opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME);
if (op->d.fetch.known_desc != NULL)
{
@@ -2991,7 +3101,9 @@ ExecComputeSlotInfo(ExprState *state, ExprEvalStep *op)
desc = ExecGetResultType(os);
}
}
- else if (opcode == EEOP_SCAN_FETCHSOME)
+ else if (opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME)
{
desc = parent->scandesc;
@@ -3039,6 +3151,12 @@ ExecInitWholeRowVar(ExprEvalStep *scratch, Var *variable, ExprState *state)
scratch->d.wholerow.tupdesc = NULL; /* filled at runtime */
scratch->d.wholerow.junkFilter = NULL;
+ /* update ExprState flags if Var refers to OLD/NEW */
+ if (variable->varreturningtype == VAR_RETURNING_OLD)
+ state->flags |= EEO_FLAG_HAS_OLD;
+ else if (variable->varreturningtype == VAR_RETURNING_NEW)
+ state->flags |= EEO_FLAG_HAS_NEW;
+
/*
* If the input tuple came from a subquery, it might contain "resjunk"
* columns (such as GROUP BY or ORDER BY columns), which we don't want to
@@ -3541,7 +3659,7 @@ ExecBuildAggTrans(AggState *aggstate, AggStatePerPhase phase,
PlanState *parent = &aggstate->ss.ps;
ExprEvalStep scratch = {0};
bool isCombine = DO_AGGSPLIT_COMBINE(aggstate->aggsplit);
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
state->expr = (Expr *) aggstate;
state->parent = parent;
@@ -4082,6 +4200,7 @@ ExecBuildHash32FromAttrs(TupleDesc desc, const TupleTableSlotOps *ops,
scratch.resnull = &fcinfo->args[0].isnull;
scratch.d.var.attnum = attnum;
scratch.d.var.vartype = TupleDescAttr(desc, attnum)->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
ExprEvalPushStep(state, &scratch);
@@ -4407,6 +4526,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc, TupleDesc rdesc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = latt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4415,6 +4535,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc, TupleDesc rdesc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = ratt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4541,6 +4662,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4549,6 +4671,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index d2987663e6..45985cdc80 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -462,6 +462,8 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
/*
* This array has to be in the same order as enum ExprEvalOp.
@@ -472,16 +474,24 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
&&CASE_EEOP_INNER_FETCHSOME,
&&CASE_EEOP_OUTER_FETCHSOME,
&&CASE_EEOP_SCAN_FETCHSOME,
+ &&CASE_EEOP_OLD_FETCHSOME,
+ &&CASE_EEOP_NEW_FETCHSOME,
&&CASE_EEOP_INNER_VAR,
&&CASE_EEOP_OUTER_VAR,
&&CASE_EEOP_SCAN_VAR,
+ &&CASE_EEOP_OLD_VAR,
+ &&CASE_EEOP_NEW_VAR,
&&CASE_EEOP_INNER_SYSVAR,
&&CASE_EEOP_OUTER_SYSVAR,
&&CASE_EEOP_SCAN_SYSVAR,
+ &&CASE_EEOP_OLD_SYSVAR,
+ &&CASE_EEOP_NEW_SYSVAR,
&&CASE_EEOP_WHOLEROW,
&&CASE_EEOP_ASSIGN_INNER_VAR,
&&CASE_EEOP_ASSIGN_OUTER_VAR,
&&CASE_EEOP_ASSIGN_SCAN_VAR,
+ &&CASE_EEOP_ASSIGN_OLD_VAR,
+ &&CASE_EEOP_ASSIGN_NEW_VAR,
&&CASE_EEOP_ASSIGN_TMP,
&&CASE_EEOP_ASSIGN_TMP_MAKE_RO,
&&CASE_EEOP_CONST,
@@ -523,6 +533,7 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
&&CASE_EEOP_SQLVALUEFUNCTION,
&&CASE_EEOP_CURRENTOFEXPR,
&&CASE_EEOP_NEXTVALUEEXPR,
+ &&CASE_EEOP_RETURNINGEXPR,
&&CASE_EEOP_ARRAYEXPR,
&&CASE_EEOP_ARRAYCOERCE,
&&CASE_EEOP_ROW,
@@ -591,6 +602,8 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
#if defined(EEO_USE_COMPUTED_GOTO)
EEO_DISPATCH();
@@ -630,6 +643,24 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, oldslot);
+
+ slot_getsomeattrs(oldslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, newslot);
+
+ slot_getsomeattrs(newslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_VAR)
{
int attnum = op->d.var.attnum;
@@ -673,6 +704,32 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ *op->resvalue = oldslot->tts_values[attnum];
+ *op->resnull = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ *op->resvalue = newslot->tts_values[attnum];
+ *op->resnull = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_SYSVAR)
{
ExecEvalSysVar(state, op, econtext, innerslot);
@@ -691,6 +748,18 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, oldslot);
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, newslot);
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_WHOLEROW)
{
/* too complex for an inline implementation */
@@ -750,6 +819,40 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_ASSIGN_OLD_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = oldslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_ASSIGN_NEW_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = newslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ASSIGN_TMP)
{
int resultnum = op->d.assign_tmp.resultnum;
@@ -1438,6 +1541,23 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_RETURNINGEXPR)
+ {
+ /*
+ * The next op actually evaluates the expression. If the OLD/NEW
+ * row doesn't exist, skip that and return NULL.
+ */
+ if (state->flags & op->d.returningexpr.nullflag)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+
+ EEO_JUMP(op->d.returningexpr.jumpdone);
+ }
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ARRAYEXPR)
{
/* too complex for an inline implementation */
@@ -2119,10 +2239,14 @@ CheckExprStillValid(ExprState *state, ExprContext *econtext)
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
for (int i = 0; i < state->steps_len; i++)
{
@@ -2153,6 +2277,22 @@ CheckExprStillValid(ExprState *state, ExprContext *econtext)
CheckVarSlotCompatibility(scanslot, attnum + 1, op->d.var.vartype);
break;
}
+
+ case EEOP_OLD_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(oldslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
+
+ case EEOP_NEW_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(newslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
default:
break;
}
@@ -5113,7 +5253,7 @@ void
ExecEvalWholeRowVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext)
{
Var *variable = op->d.wholerow.var;
- TupleTableSlot *slot;
+ TupleTableSlot *slot = NULL;
TupleDesc output_tupdesc;
MemoryContext oldcontext;
HeapTupleHeader dtuple;
@@ -5138,8 +5278,40 @@ ExecEvalWholeRowVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext)
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- slot = econtext->ecxt_scantuple;
+
+ /*
+ * Get the tuple from the relation being scanned.
+ *
+ * By default, this uses the "scan" tuple slot, but a wholerow Var
+ * in the RETURNING list may explicitly refer to OLD/NEW. If the
+ * OLD/NEW row doesn't exist, we just return NULL.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ slot = econtext->ecxt_scantuple;
+ break;
+
+ case VAR_RETURNING_OLD:
+ if (state->flags & EEO_FLAG_OLD_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_oldtuple;
+ break;
+
+ case VAR_RETURNING_NEW:
+ if (state->flags & EEO_FLAG_NEW_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_newtuple;
+ break;
+ }
break;
}
@@ -5342,6 +5514,27 @@ ExecEvalSysVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext,
{
Datum d;
+ /*
+ * For OLD/NEW system attributes, check whether the OLD/NEW row exists. If
+ * it doesn't, the OLD/NEW system attribute is NULL.
+ */
+ if (op->d.var.varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ bool rowIsNull;
+
+ if (op->d.var.varreturningtype == VAR_RETURNING_OLD)
+ rowIsNull = (state->flags & EEO_FLAG_OLD_IS_NULL) != 0;
+ else
+ rowIsNull = (state->flags & EEO_FLAG_NEW_IS_NULL) != 0;
+
+ if (rowIsNull)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ }
+
/* slot_getsysattr has sufficient defenses against bad attnums */
d = slot_getsysattr(slot,
op->d.var.attnum,
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
index 1c12d6ebff..e7be7345f6 100644
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -1257,6 +1257,7 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo,
resultRelInfo->ri_ReturningSlot = NULL;
resultRelInfo->ri_TrigOldSlot = NULL;
resultRelInfo->ri_TrigNewSlot = NULL;
+ resultRelInfo->ri_AllNullSlot = NULL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_MATCHED] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_SOURCE] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_TARGET] = NIL;
diff --git a/src/backend/executor/execUtils.c b/src/backend/executor/execUtils.c
index df0223129c..f7738a1a10 100644
--- a/src/backend/executor/execUtils.c
+++ b/src/backend/executor/execUtils.c
@@ -1242,6 +1242,34 @@ ExecGetReturningSlot(EState *estate, ResultRelInfo *relInfo)
return relInfo->ri_ReturningSlot;
}
+/*
+ * Return a relInfo's all-NULL tuple slot for processing returning tuples.
+ *
+ * Note: this slot is intentionally filled with NULLs in every column, and
+ * should be considered read-only --- the caller must not update it.
+ */
+TupleTableSlot *
+ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo)
+{
+ if (relInfo->ri_AllNullSlot == NULL)
+ {
+ Relation rel = relInfo->ri_RelationDesc;
+ MemoryContext oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
+ TupleTableSlot *slot;
+
+ slot = ExecInitExtraTupleSlot(estate,
+ RelationGetDescr(rel),
+ table_slot_callbacks(rel));
+ ExecStoreAllNullTuple(slot);
+
+ relInfo->ri_AllNullSlot = slot;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ return relInfo->ri_AllNullSlot;
+}
+
/*
* Return the map needed to convert given child result relation's tuples to
* the rowtype of the query's main target ("root") relation. Note that a
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index c445c433df..aac99faae3 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -101,6 +101,13 @@ typedef struct ModifyTableContext
*/
TM_FailureData tmfd;
+ /*
+ * The tuple deleted when doing a cross-partition UPDATE with a RETURNING
+ * clause that refers to OLD columns (converted to the root's tuple
+ * descriptor).
+ */
+ TupleTableSlot *cpDeletedSlot;
+
/*
* The tuple projected by the INSERT's RETURNING clause, when doing a
* cross-partition UPDATE
@@ -243,34 +250,81 @@ ExecCheckPlanOutput(Relation resultRel, List *targetList)
/*
* ExecProcessReturning --- evaluate a RETURNING list
*
+ * context: context for the ModifyTable operation
* resultRelInfo: current result rel
- * tupleSlot: slot holding tuple actually inserted/updated/deleted
+ * cmdType: operation/merge action performed (INSERT, UPDATE, or DELETE)
+ * oldSlot: slot holding old tuple deleted or updated
+ * newSlot: slot holding new tuple inserted or updated
* planSlot: slot holding tuple returned by top subplan node
*
- * Note: If tupleSlot is NULL, the FDW should have already provided econtext's
- * scan tuple.
+ * Note: If oldSlot and newSlot are NULL, the FDW should have already provided
+ * econtext's scan tuple and its old & new tuples are not needed (FDW direct-
+ * modify is disabled if the RETURNING list refers to any OLD/NEW values).
*
* Returns a slot holding the result tuple
*/
static TupleTableSlot *
-ExecProcessReturning(ResultRelInfo *resultRelInfo,
- TupleTableSlot *tupleSlot,
+ExecProcessReturning(ModifyTableContext *context,
+ ResultRelInfo *resultRelInfo,
+ CmdType cmdType,
+ TupleTableSlot *oldSlot,
+ TupleTableSlot *newSlot,
TupleTableSlot *planSlot)
{
+ EState *estate = context->estate;
ProjectionInfo *projectReturning = resultRelInfo->ri_projectReturning;
ExprContext *econtext = projectReturning->pi_exprContext;
/* Make tuple and any needed join variables available to ExecProject */
- if (tupleSlot)
- econtext->ecxt_scantuple = tupleSlot;
+ switch (cmdType)
+ {
+ case CMD_INSERT:
+ case CMD_UPDATE:
+ /* return new tuple by default */
+ if (newSlot)
+ econtext->ecxt_scantuple = newSlot;
+ break;
+
+ case CMD_DELETE:
+ /* return old tuple by default */
+ if (oldSlot)
+ econtext->ecxt_scantuple = oldSlot;
+ break;
+
+ default:
+ elog(ERROR, "unrecognized commandType: %d", (int) cmdType);
+ }
econtext->ecxt_outertuple = planSlot;
+ /* Make old/new tuples available to ExecProject, if required */
+ if (oldSlot)
+ econtext->ecxt_oldtuple = oldSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ econtext->ecxt_oldtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_oldtuple = NULL; /* No references to OLD columns */
+
+ if (newSlot)
+ econtext->ecxt_newtuple = newSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW)
+ econtext->ecxt_newtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_newtuple = NULL; /* No references to NEW columns */
+
/*
- * RETURNING expressions might reference the tableoid column, so
- * reinitialize tts_tableOid before evaluating them.
+ * Tell ExecProject whether or not the OLD/NEW rows actually exist. This
+ * information is required to evaluate ReturningExpr nodes and also in
+ * ExecEvalSysVar() and ExecEvalWholeRowVar().
*/
- econtext->ecxt_scantuple->tts_tableOid =
- RelationGetRelid(resultRelInfo->ri_RelationDesc);
+ if (oldSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_OLD_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_OLD_IS_NULL;
+
+ if (newSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_NEW_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_NEW_IS_NULL;
/* Compute the RETURNING expressions */
return ExecProject(projectReturning);
@@ -1204,7 +1258,56 @@ ExecInsert(ModifyTableContext *context,
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- result = ExecProcessReturning(resultRelInfo, slot, planSlot);
+ {
+ TupleTableSlot *oldSlot = NULL;
+
+ /*
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD columns, ExecDelete() will have saved the tuple
+ * deleted from the original partition, which we must use here to
+ * compute the OLD column values. Otherwise, all OLD column values
+ * will be NULL.
+ */
+ if (context->cpDeletedSlot)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the OLD tuple to the new partition's format/slot, if
+ * needed. Note that ExceDelete() already converted it to the
+ * root's partition's format/slot.
+ */
+ oldSlot = context->cpDeletedSlot;
+ tupconv_map = ExecGetRootToChildMap(resultRelInfo, estate);
+ if (tupconv_map != NULL)
+ {
+ oldSlot = execute_attr_map_slot(tupconv_map->attrMap,
+ oldSlot,
+ ExecGetReturningSlot(estate,
+ resultRelInfo));
+
+ oldSlot->tts_tableOid = context->cpDeletedSlot->tts_tableOid;
+ ItemPointerCopy(&context->cpDeletedSlot->tts_tid, &oldSlot->tts_tid);
+ }
+ }
+
+ result = ExecProcessReturning(context, resultRelInfo, CMD_INSERT,
+ oldSlot, slot, planSlot);
+
+ /*
+ * For a cross-partition UPDATE, release the old tuple, first making
+ * sure that the result slot has a local copy of any pass-by-reference
+ * values.
+ */
+ if (context->cpDeletedSlot)
+ {
+ ExecMaterializeSlot(result);
+ ExecClearTuple(oldSlot);
+ if (context->cpDeletedSlot != oldSlot)
+ ExecClearTuple(context->cpDeletedSlot);
+ context->cpDeletedSlot = NULL;
+ }
+ }
if (inserted_tuple)
*inserted_tuple = slot;
@@ -1442,6 +1545,7 @@ ExecDelete(ModifyTableContext *context,
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
TupleTableSlot *slot = NULL;
TM_Result result;
+ bool saveOld;
if (tupleDeleted)
*tupleDeleted = false;
@@ -1676,8 +1780,17 @@ ldelete:
ExecDeleteEpilogue(context, resultRelInfo, tupleid, oldtuple, changingPart);
- /* Process RETURNING if present and if requested */
- if (processReturning && resultRelInfo->ri_projectReturning)
+ /*
+ * Process RETURNING if present and if requested.
+ *
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD column values, save the old tuple here for later
+ * processing of the RETURNING list by ExecInsert().
+ */
+ saveOld = changingPart && resultRelInfo->ri_projectReturning &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD;
+
+ if (resultRelInfo->ri_projectReturning && (processReturning || saveOld))
{
/*
* We have to put the target tuple into a slot, which means first we
@@ -1705,7 +1818,41 @@ ldelete:
}
}
- rslot = ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ /*
+ * If required, save the old tuple for later processing of the
+ * RETURNING list by ExecInsert().
+ */
+ if (saveOld)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the tuple into the root partition's format/slot, if
+ * needed. ExecInsert() will then convert it to the new
+ * partition's format/slot, if necessary.
+ */
+ tupconv_map = ExecGetChildToRootMap(resultRelInfo);
+ if (tupconv_map != NULL)
+ {
+ ResultRelInfo *rootRelInfo = context->mtstate->rootResultRelInfo;
+ TupleTableSlot *oldSlot = slot;
+
+ slot = execute_attr_map_slot(tupconv_map->attrMap,
+ slot,
+ ExecGetReturningSlot(estate,
+ rootRelInfo));
+
+ slot->tts_tableOid = oldSlot->tts_tableOid;
+ ItemPointerCopy(&oldSlot->tts_tid, &slot->tts_tid);
+ }
+
+ context->cpDeletedSlot = slot;
+
+ return NULL;
+ }
+
+ rslot = ExecProcessReturning(context, resultRelInfo, CMD_DELETE,
+ slot, NULL, context->planSlot);
/*
* Before releasing the target tuple again, make sure rslot has a
@@ -1758,6 +1905,7 @@ ExecCrossPartitionUpdate(ModifyTableContext *context,
bool tuple_deleted;
TupleTableSlot *epqslot = NULL;
+ context->cpDeletedSlot = NULL;
context->cpUpdateReturningSlot = NULL;
*retry_slot = NULL;
@@ -2258,6 +2406,7 @@ ExecCrossPartitionUpdateForeignKey(ModifyTableContext *context,
* the planSlot. oldtuple is passed to foreign table triggers; it is
* NULL when the foreign table has no relevant triggers.
*
+ * oldSlot contains the old tuple value.
* slot contains the new tuple value to be stored.
* planSlot is the output of the ModifyTable's subplan; we use it
* to access values from other input tables (for RETURNING),
@@ -2270,8 +2419,8 @@ ExecCrossPartitionUpdateForeignKey(ModifyTableContext *context,
*/
static TupleTableSlot *
ExecUpdate(ModifyTableContext *context, ResultRelInfo *resultRelInfo,
- ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot,
- bool canSetTag)
+ ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *oldSlot,
+ TupleTableSlot *slot, bool canSetTag)
{
EState *estate = context->estate;
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
@@ -2389,7 +2538,6 @@ redo_act:
{
TupleTableSlot *inputslot;
TupleTableSlot *epqslot;
- TupleTableSlot *oldSlot;
if (IsolationUsesXactSnapshot())
ereport(ERROR,
@@ -2504,7 +2652,8 @@ redo_act:
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- return ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ return ExecProcessReturning(context, resultRelInfo, CMD_UPDATE,
+ oldSlot, slot, context->planSlot);
return NULL;
}
@@ -2724,16 +2873,23 @@ ExecOnConflictUpdate(ModifyTableContext *context,
/* Execute UPDATE with projection */
*returning = ExecUpdate(context, resultRelInfo,
- conflictTid, NULL,
+ conflictTid, NULL, existing,
resultRelInfo->ri_onConflict->oc_ProjSlot,
canSetTag);
/*
* Clear out existing tuple, as there might not be another conflict among
* the next input rows. Don't want to hold resources till the end of the
- * query.
+ * query. First though, make sure that the returning slot, if any, has a
+ * local copy of any OLD pass-by-reference values, if it refers to any OLD
+ * columns.
*/
+ if (*returning != NULL &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ ExecMaterializeSlot(*returning);
+
ExecClearTuple(existing);
+
return true;
}
@@ -3338,13 +3494,20 @@ lmerge_matched:
switch (commandType)
{
case CMD_UPDATE:
- rslot = ExecProcessReturning(resultRelInfo, newslot,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_UPDATE,
+ resultRelInfo->ri_oldTupleSlot,
+ newslot,
context->planSlot);
break;
case CMD_DELETE:
- rslot = ExecProcessReturning(resultRelInfo,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_DELETE,
resultRelInfo->ri_oldTupleSlot,
+ NULL,
context->planSlot);
break;
@@ -3894,6 +4057,7 @@ ExecModifyTable(PlanState *pstate)
if (node->mt_merge_pending_not_matched != NULL)
{
context.planSlot = node->mt_merge_pending_not_matched;
+ context.cpDeletedSlot = NULL;
slot = ExecMergeNotMatched(&context, node->resultRelInfo,
node->canSetTag);
@@ -3913,6 +4077,7 @@ ExecModifyTable(PlanState *pstate)
/* Fetch the next row from subplan */
context.planSlot = ExecProcNode(subplanstate);
+ context.cpDeletedSlot = NULL;
/* No more tuples to process? */
if (TupIsNull(context.planSlot))
@@ -3980,9 +4145,15 @@ ExecModifyTable(PlanState *pstate)
* A scan slot containing the data that was actually inserted,
* updated or deleted has already been made available to
* ExecProcessReturning by IterateDirectModify, so no need to
- * provide it here.
+ * provide it here. The individual old and new slots are not
+ * needed, since direct-modify is disabled if the RETURNING list
+ * refers to OLD/NEW values.
*/
- slot = ExecProcessReturning(resultRelInfo, NULL, context.planSlot);
+ Assert((resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD) == 0 &&
+ (resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW) == 0);
+
+ slot = ExecProcessReturning(&context, resultRelInfo, operation,
+ NULL, NULL, context.planSlot);
return slot;
}
@@ -4172,7 +4343,7 @@ ExecModifyTable(PlanState *pstate)
/* Now apply the update. */
slot = ExecUpdate(&context, resultRelInfo, tupleid, oldtuple,
- slot, node->canSetTag);
+ oldSlot, slot, node->canSetTag);
if (tuplock)
UnlockTuple(resultRelInfo->ri_RelationDesc, tupleid,
InplaceUpdateTupleLock);
diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c
index c533f55254..9a10e6f47b 100644
--- a/src/backend/jit/llvm/llvmjit_expr.c
+++ b/src/backend/jit/llvm/llvmjit_expr.c
@@ -105,6 +105,8 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_innerslot;
LLVMValueRef v_outerslot;
LLVMValueRef v_scanslot;
+ LLVMValueRef v_oldslot;
+ LLVMValueRef v_newslot;
LLVMValueRef v_resultslot;
/* nulls/values of slots */
@@ -114,6 +116,10 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_outernulls;
LLVMValueRef v_scanvalues;
LLVMValueRef v_scannulls;
+ LLVMValueRef v_oldvalues;
+ LLVMValueRef v_oldnulls;
+ LLVMValueRef v_newvalues;
+ LLVMValueRef v_newnulls;
LLVMValueRef v_resultvalues;
LLVMValueRef v_resultnulls;
@@ -200,6 +206,16 @@ llvm_compile_expr(ExprState *state)
v_econtext,
FIELDNO_EXPRCONTEXT_OUTERTUPLE,
"v_outerslot");
+ v_oldslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_OLDTUPLE,
+ "v_oldslot");
+ v_newslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_NEWTUPLE,
+ "v_newslot");
v_resultslot = l_load_struct_gep(b,
StructExprState,
v_state,
@@ -237,6 +253,26 @@ llvm_compile_expr(ExprState *state)
v_outerslot,
FIELDNO_TUPLETABLESLOT_ISNULL,
"v_outernulls");
+ v_oldvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_oldvalues");
+ v_oldnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_oldnulls");
+ v_newvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_newvalues");
+ v_newnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_newnulls");
v_resultvalues = l_load_struct_gep(b,
StructTupleTableSlot,
v_resultslot,
@@ -302,6 +338,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_FETCHSOME:
case EEOP_OUTER_FETCHSOME:
case EEOP_SCAN_FETCHSOME:
+ case EEOP_OLD_FETCHSOME:
+ case EEOP_NEW_FETCHSOME:
{
TupleDesc desc = NULL;
LLVMValueRef v_slot;
@@ -326,8 +364,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_FETCHSOME)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_FETCHSOME)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_FETCHSOME)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
/*
* Check if all required attributes are available, or
@@ -396,6 +438,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_VAR:
case EEOP_OUTER_VAR:
case EEOP_SCAN_VAR:
+ case EEOP_OLD_VAR:
+ case EEOP_NEW_VAR:
{
LLVMValueRef value,
isnull;
@@ -413,11 +457,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
v_attnum = l_int32_const(lc, op->d.var.attnum);
value = l_load_gep1(b, TypeSizeT, v_values, v_attnum, "");
@@ -432,6 +486,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_SYSVAR:
case EEOP_OUTER_SYSVAR:
case EEOP_SCAN_SYSVAR:
+ case EEOP_OLD_SYSVAR:
+ case EEOP_NEW_SYSVAR:
{
LLVMValueRef v_slot;
@@ -439,8 +495,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_SYSVAR)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_SYSVAR)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_SYSVAR)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
build_EvalXFunc(b, mod, "ExecEvalSysVar",
v_state, op, v_econtext, v_slot);
@@ -458,6 +518,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_ASSIGN_INNER_VAR:
case EEOP_ASSIGN_OUTER_VAR:
case EEOP_ASSIGN_SCAN_VAR:
+ case EEOP_ASSIGN_OLD_VAR:
+ case EEOP_ASSIGN_NEW_VAR:
{
LLVMValueRef v_value;
LLVMValueRef v_isnull;
@@ -478,11 +540,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_ASSIGN_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_ASSIGN_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
/* load data */
v_attnum = l_int32_const(lc, op->d.assign_var.attnum);
@@ -1654,6 +1726,45 @@ llvm_compile_expr(ExprState *state)
LLVMBuildBr(b, opblocks[opno + 1]);
break;
+ case EEOP_RETURNINGEXPR:
+ {
+ LLVMBasicBlockRef b_isnull;
+ LLVMValueRef v_flagsp;
+ LLVMValueRef v_flags;
+ LLVMValueRef v_nullflag;
+
+ b_isnull = l_bb_before_v(opblocks[opno + 1],
+ "op.%d.row.isnull", opno);
+
+ /*
+ * The next op actually evaluates the expression. If the
+ * OLD/NEW row doesn't exist, skip that and return NULL.
+ */
+ v_flagsp = l_struct_gep(b,
+ StructExprState,
+ v_state,
+ FIELDNO_EXPRSTATE_FLAGS,
+ "v.state.flags");
+ v_flags = l_load(b, TypeStorageBool, v_flagsp, "");
+
+ v_nullflag = l_int8_const(lc, op->d.returningexpr.nullflag);
+
+ LLVMBuildCondBr(b,
+ LLVMBuildICmp(b, LLVMIntEQ,
+ LLVMBuildAnd(b, v_flags,
+ v_nullflag, ""),
+ l_sbool_const(0), ""),
+ opblocks[opno + 1], b_isnull);
+
+ LLVMPositionBuilderAtEnd(b, b_isnull);
+
+ LLVMBuildStore(b, l_sizet_const(0), v_resvaluep);
+ LLVMBuildStore(b, l_sbool_const(1), v_resnullp);
+
+ LLVMBuildBr(b, opblocks[op->d.returningexpr.jumpdone]);
+ break;
+ }
+
case EEOP_ARRAYEXPR:
build_EvalXFunc(b, mod, "ExecEvalArrayExpr",
v_state, op);
diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c
index 7e5df7bea4..0662ee0d44 100644
--- a/src/backend/nodes/makefuncs.c
+++ b/src/backend/nodes/makefuncs.c
@@ -80,12 +80,14 @@ makeVar(int varno,
var->varlevelsup = varlevelsup;
/*
- * Only a few callers need to make Var nodes with non-null varnullingrels,
- * or with varnosyn/varattnosyn different from varno/varattno. We don't
- * provide separate arguments for them, but just initialize them to NULL
- * and the given varno/varattno. This reduces code clutter and chance of
- * error for most callers.
+ * Only a few callers need to make Var nodes with varreturningtype
+ * different from VAR_RETURNING_DEFAULT, non-null varnullingrels, or with
+ * varnosyn/varattnosyn different from varno/varattno. We don't provide
+ * separate arguments for them, but just initialize them to sensible
+ * default values. This reduces code clutter and chance of error for most
+ * callers.
*/
+ var->varreturningtype = VAR_RETURNING_DEFAULT;
var->varnullingrels = NULL;
var->varnosyn = (Index) varno;
var->varattnosyn = varattno;
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 50705a1e15..e4dda335fb 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -278,6 +278,9 @@ exprType(const Node *expr)
type = exprType((Node *) n->expr);
}
break;
+ case T_ReturningExpr:
+ type = exprType((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
type = exprType((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -529,6 +532,8 @@ exprTypmod(const Node *expr)
return ((const CoerceToDomainValue *) expr)->typeMod;
case T_SetToDefault:
return ((const SetToDefault *) expr)->typeMod;
+ case T_ReturningExpr:
+ return exprTypmod((Node *) ((const ReturningExpr *) expr)->retexpr);
case T_PlaceHolderVar:
return exprTypmod((Node *) ((const PlaceHolderVar *) expr)->phexpr);
default:
@@ -1047,6 +1052,9 @@ exprCollation(const Node *expr)
case T_InferenceElem:
coll = exprCollation((Node *) ((const InferenceElem *) expr)->expr);
break;
+ case T_ReturningExpr:
+ coll = exprCollation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
coll = exprCollation((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -1298,6 +1306,10 @@ exprSetCollation(Node *expr, Oid collation)
/* NextValueExpr's result is an integer type ... */
Assert(!OidIsValid(collation)); /* ... so never set a collation */
break;
+ case T_ReturningExpr:
+ exprSetCollation((Node *) ((ReturningExpr *) expr)->retexpr,
+ collation);
+ break;
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(expr));
break;
@@ -1624,6 +1636,9 @@ exprLocation(const Node *expr)
case T_SetToDefault:
loc = ((const SetToDefault *) expr)->location;
break;
+ case T_ReturningExpr:
+ loc = exprLocation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_TargetEntry:
/* just use argument's location */
loc = exprLocation((Node *) ((const TargetEntry *) expr)->expr);
@@ -2613,6 +2628,8 @@ expression_tree_walker_impl(Node *node,
return WALK(((PlaceHolderVar *) node)->phexpr);
case T_InferenceElem:
return WALK(((InferenceElem *) node)->expr);
+ case T_ReturningExpr:
+ return WALK(((ReturningExpr *) node)->retexpr);
case T_AppendRelInfo:
{
AppendRelInfo *appinfo = (AppendRelInfo *) node;
@@ -3454,6 +3471,16 @@ expression_tree_mutator_impl(Node *node,
return (Node *) newnode;
}
break;
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ ReturningExpr *newnode;
+
+ FLATCOPY(newnode, rexpr, ReturningExpr);
+ MUTATE(newnode->retexpr, rexpr->retexpr, Expr *);
+ return (Node *) newnode;
+ }
+ break;
case T_TargetEntry:
{
TargetEntry *targetentry = (TargetEntry *) node;
@@ -4005,6 +4032,7 @@ raw_expression_tree_walker_impl(Node *node,
case T_A_Const:
case T_A_Star:
case T_MergeSupportFunc:
+ case T_ReturningOption:
/* primitive node types with no subnodes */
break;
case T_Alias:
@@ -4233,7 +4261,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->onConflictClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4249,7 +4277,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->whereClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4267,7 +4295,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->fromClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4285,7 +4313,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->mergeWhenClauses))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4303,6 +4331,16 @@ raw_expression_tree_walker_impl(Node *node,
return true;
}
break;
+ case T_ReturningClause:
+ {
+ ReturningClause *returning = (ReturningClause *) node;
+
+ if (WALK(returning->options))
+ return true;
+ if (WALK(returning->exprs))
+ return true;
+ }
+ break;
case T_SelectStmt:
{
SelectStmt *stmt = (SelectStmt *) node;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 172edb643a..6346c4e60f 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -3985,6 +3985,7 @@ subquery_push_qual(Query *subquery, RangeTblEntry *rte, Index rti, Node *qual)
*/
qual = ReplaceVarsFromTargetList(qual, rti, 0, rte,
subquery->targetList,
+ subquery->resultRelation,
REPLACEVARS_REPORT_ERROR, 0,
&subquery->hasSubLinks);
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index b3e2294e84..bf32837269 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -7121,6 +7121,8 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
int epqParam)
{
ModifyTable *node = makeNode(ModifyTable);
+ bool returning_old_or_new = false;
+ bool returning_old_or_new_valid = false;
List *fdw_private_list;
Bitmapset *direct_modify_plans;
ListCell *lc;
@@ -7185,6 +7187,8 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
}
node->updateColnosLists = updateColnosLists;
node->withCheckOptionLists = withCheckOptionLists;
+ node->returningOldAlias = root->parse->returningOldAlias;
+ node->returningNewAlias = root->parse->returningNewAlias;
node->returningLists = returningLists;
node->rowMarks = rowMarks;
node->mergeActionLists = mergeActionLists;
@@ -7265,7 +7269,8 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
* callback functions needed for that and (2) there are no local
* structures that need to be run for each modified row: row-level
* triggers on the foreign table, stored generated columns, WITH CHECK
- * OPTIONs from parent views.
+ * OPTIONs from parent views, or Vars returning OLD/NEW in the
+ * RETURNING list.
*/
direct_modify = false;
if (fdwroutine != NULL &&
@@ -7276,7 +7281,18 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
withCheckOptionLists == NIL &&
!has_row_triggers(root, rti, operation) &&
!has_stored_generated_columns(root, rti))
- direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ {
+ /* returning_old_or_new is the same for all result relations */
+ if (!returning_old_or_new_valid)
+ {
+ returning_old_or_new =
+ contain_vars_returning_old_or_new((Node *)
+ root->parse->returningList);
+ returning_old_or_new_valid = true;
+ }
+ if (!returning_old_or_new)
+ direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ }
if (direct_modify)
direct_modify_plans = bms_add_member(direct_modify_plans, i);
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 6d23df108d..7c67758308 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -3070,6 +3070,21 @@ fix_join_expr_mutator(Node *node, fix_join_expr_context *context)
{
Var *var = (Var *) node;
+ /*
+ * Verify that Vars with non-default varreturningtype only appear in
+ * the RETURNING list, and refer to the target relation.
+ */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ if (context->inner_itlist != NULL ||
+ context->outer_itlist == NULL ||
+ context->acceptable_rel == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+ if (var->varno != context->acceptable_rel)
+ elog(ERROR, "wrong varno %d (expected %d) for variable returning old/new",
+ var->varno, context->acceptable_rel);
+ }
+
/* Look for the var in the input tlists, first in the outer */
if (context->outer_itlist)
{
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index ed62e3a0fc..418d89f2fc 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -354,17 +354,19 @@ build_subplan(PlannerInfo *root, Plan *plan, Path *path,
Node *arg = pitem->item;
/*
- * The Var, PlaceHolderVar, Aggref or GroupingFunc has already been
- * adjusted to have the correct varlevelsup, phlevelsup, or
- * agglevelsup.
+ * The Var, PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr has
+ * already been adjusted to have the correct varlevelsup, phlevelsup,
+ * agglevelsup, or retlevelsup.
*
- * If it's a PlaceHolderVar, Aggref or GroupingFunc, its arguments
- * might contain SubLinks, which have not yet been processed (see the
- * comments for SS_replace_correlation_vars). Do that now.
+ * If it's a PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr,
+ * its arguments might contain SubLinks, which have not yet been
+ * processed (see the comments for SS_replace_correlation_vars). Do
+ * that now.
*/
if (IsA(arg, PlaceHolderVar) ||
IsA(arg, Aggref) ||
- IsA(arg, GroupingFunc))
+ IsA(arg, GroupingFunc) ||
+ IsA(arg, ReturningExpr))
arg = SS_process_sublinks(root, arg, false);
splan->parParam = lappend_int(splan->parParam, pitem->paramId);
@@ -1863,8 +1865,8 @@ convert_EXISTS_to_ANY(PlannerInfo *root, Query *subselect,
/*
* Replace correlation vars (uplevel vars) with Params.
*
- * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions, and
- * MergeSupportFuncs are replaced, too.
+ * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions,
+ * MergeSupportFuncs, and ReturningExprs are replaced, too.
*
* Note: it is critical that this runs immediately after SS_process_sublinks.
* Since we do not recurse into the arguments of uplevel PHVs and aggregates,
@@ -1924,6 +1926,12 @@ replace_correlation_vars_mutator(Node *node, PlannerInfo *root)
return (Node *) replace_outer_merge_support(root,
(MergeSupportFunc *) node);
}
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return (Node *) replace_outer_returning(root,
+ (ReturningExpr *) node);
+ }
return expression_tree_mutator(node, replace_correlation_vars_mutator, root);
}
@@ -1977,11 +1985,11 @@ process_sublinks_mutator(Node *node, process_sublinks_context *context)
}
/*
- * Don't recurse into the arguments of an outer PHV, Aggref or
- * GroupingFunc here. Any SubLinks in the arguments have to be dealt with
- * at the outer query level; they'll be handled when build_subplan
- * collects the PHV, Aggref or GroupingFunc into the arguments to be
- * passed down to the current subplan.
+ * Don't recurse into the arguments of an outer PHV, Aggref, GroupingFunc
+ * or ReturningExpr here. Any SubLinks in the arguments have to be dealt
+ * with at the outer query level; they'll be handled when build_subplan
+ * collects the PHV, Aggref, GroupingFunc or ReturningExpr into the
+ * arguments to be passed down to the current subplan.
*/
if (IsA(node, PlaceHolderVar))
{
@@ -1998,6 +2006,11 @@ process_sublinks_mutator(Node *node, process_sublinks_context *context)
if (((GroupingFunc *) node)->agglevelsup > 0)
return node;
}
+ else if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return node;
+ }
/*
* We should never see a SubPlan expression in the input (since this is
@@ -2110,7 +2123,9 @@ SS_identify_outer_params(PlannerInfo *root)
outer_params = NULL;
for (proot = root->parent_root; proot != NULL; proot = proot->parent_root)
{
- /* Include ordinary Var/PHV/Aggref/GroupingFunc params */
+ /*
+ * Include ordinary Var/PHV/Aggref/GroupingFunc/ReturningExpr params.
+ */
foreach(l, proot->plan_params)
{
PlannerParamItem *pitem = (PlannerParamItem *) lfirst(l);
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
index adad7ea9a9..901938e8b4 100644
--- a/src/backend/optimizer/prep/prepjointree.c
+++ b/src/backend/optimizer/prep/prepjointree.c
@@ -2539,7 +2539,8 @@ pullup_replace_vars_callback(Var *var,
* expansion with varlevelsup = 0, and then adjust below if needed.
*/
expandRTE(rcon->target_rte,
- var->varno, 0 /* not varlevelsup */ , var->location,
+ var->varno, 0 /* not varlevelsup */ ,
+ var->varreturningtype, var->location,
(var->vartype != RECORDOID),
&colnames, &fields);
/* Expand the generated per-field Vars, but don't insert PHVs there */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index 45e8b74f94..7bc1e6f246 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -253,6 +253,13 @@ adjust_appendrel_attrs_mutator(Node *node,
* all non-Var outputs of such subqueries, and then we could look up
* the pre-existing PHV here. Or perhaps just wrap the translations
* that way to begin with?
+ *
+ * If var->varreturningtype is not VAR_RETURNING_DEFAULT, then that
+ * also needs to be copied to the translated Var. That too would fail
+ * if the translation wasn't a Var, but that should never happen since
+ * a non-default var->varreturningtype is only used for Vars referring
+ * to the result relation, which should never be a flattened UNION ALL
+ * subquery.
*/
for (cnt = 0; cnt < nappinfos; cnt++)
@@ -283,9 +290,17 @@ adjust_appendrel_attrs_mutator(Node *node,
elog(ERROR, "attribute %d of relation \"%s\" does not exist",
var->varattno, get_rel_name(appinfo->parent_reloid));
if (IsA(newnode, Var))
+ {
+ ((Var *) newnode)->varreturningtype = var->varreturningtype;
((Var *) newnode)->varnullingrels = var->varnullingrels;
- else if (var->varnullingrels != NULL)
- elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
+ else
+ {
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
+ if (var->varnullingrels != NULL)
+ elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
return newnode;
}
else if (var->varattno == 0)
@@ -339,6 +354,8 @@ adjust_appendrel_attrs_mutator(Node *node,
rowexpr->colnames = copyObject(rte->eref->colnames);
rowexpr->location = -1;
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
if (var->varnullingrels != NULL)
elog(ERROR, "failed to apply nullingrels to a non-Var");
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index bb7a9b7728..91a0b84890 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -1295,6 +1295,7 @@ contain_leaked_vars_walker(Node *node, void *context)
case T_NullTest:
case T_BooleanTest:
case T_NextValueExpr:
+ case T_ReturningExpr:
case T_List:
/*
@@ -3404,6 +3405,8 @@ eval_const_expressions_mutator(Node *node,
fselect->resulttypmod,
fselect->resultcollid,
((Var *) arg)->varlevelsup);
+ /* New Var has same OLD/NEW returning as old one */
+ newvar->varreturningtype = ((Var *) arg)->varreturningtype;
/* New Var is nullable by same rels as the old one */
newvar->varnullingrels = ((Var *) arg)->varnullingrels;
return (Node *) newvar;
diff --git a/src/backend/optimizer/util/paramassign.c b/src/backend/optimizer/util/paramassign.c
index f461fedf19..38a39867dd 100644
--- a/src/backend/optimizer/util/paramassign.c
+++ b/src/backend/optimizer/util/paramassign.c
@@ -91,6 +91,7 @@ assign_param_for_var(PlannerInfo *root, Var *var)
pvar->vartype == var->vartype &&
pvar->vartypmod == var->vartypmod &&
pvar->varcollid == var->varcollid &&
+ pvar->varreturningtype == var->varreturningtype &&
bms_equal(pvar->varnullingrels, var->varnullingrels))
return pitem->paramId;
}
@@ -358,6 +359,52 @@ replace_outer_merge_support(PlannerInfo *root, MergeSupportFunc *msf)
return retval;
}
+/*
+ * Generate a Param node to replace the given ReturningExpr expression which
+ * is expected to have retlevelsup > 0 (ie, it is not local). Record the need
+ * for the ReturningExpr in the proper upper-level root->plan_params.
+ */
+Param *
+replace_outer_returning(PlannerInfo *root, ReturningExpr *rexpr)
+{
+ Param *retval;
+ PlannerParamItem *pitem;
+ Index levelsup;
+ Oid ptype = exprType((Node *) rexpr->retexpr);
+
+ Assert(rexpr->retlevelsup > 0 && rexpr->retlevelsup < root->query_level);
+
+ /* Find the query level the ReturningExpr belongs to */
+ for (levelsup = rexpr->retlevelsup; levelsup > 0; levelsup--)
+ root = root->parent_root;
+
+ /*
+ * It does not seem worthwhile to try to de-duplicate references to outer
+ * ReturningExprs. Just make a new slot every time.
+ */
+ rexpr = copyObject(rexpr);
+ IncrementVarSublevelsUp((Node *) rexpr, -((int) rexpr->retlevelsup), 0);
+ Assert(rexpr->retlevelsup == 0);
+
+ pitem = makeNode(PlannerParamItem);
+ pitem->item = (Node *) rexpr;
+ pitem->paramId = list_length(root->glob->paramExecTypes);
+ root->glob->paramExecTypes = lappend_oid(root->glob->paramExecTypes,
+ ptype);
+
+ root->plan_params = lappend(root->plan_params, pitem);
+
+ retval = makeNode(Param);
+ retval->paramkind = PARAM_EXEC;
+ retval->paramid = pitem->paramId;
+ retval->paramtype = ptype;
+ retval->paramtypmod = exprTypmod((Node *) rexpr->retexpr);
+ retval->paramcollid = exprCollation((Node *) rexpr->retexpr);
+ retval->location = exprLocation((Node *) rexpr->retexpr);
+
+ return retval;
+}
+
/*
* Generate a Param node to replace the given Var,
* which is expected to come from some upper NestLoop plan node.
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index c31cc3ee69..0475b873bf 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1846,8 +1846,8 @@ build_physical_tlist(PlannerInfo *root, RelOptInfo *rel)
case RTE_NAMEDTUPLESTORE:
case RTE_RESULT:
/* Not all of these can have dropped cols, but share code anyway */
- expandRTE(rte, varno, 0, -1, true /* include dropped */ ,
- NULL, &colvars);
+ expandRTE(rte, varno, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , NULL, &colvars);
foreach(l, colvars)
{
var = (Var *) lfirst(l);
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
index 5f721eb8e1..3be079068e 100644
--- a/src/backend/optimizer/util/var.c
+++ b/src/backend/optimizer/util/var.c
@@ -76,6 +76,7 @@ static bool pull_varattnos_walker(Node *node, pull_varattnos_context *context);
static bool pull_vars_walker(Node *node, pull_vars_context *context);
static bool contain_var_clause_walker(Node *node, void *context);
static bool contain_vars_of_level_walker(Node *node, int *sublevels_up);
+static bool contain_vars_returning_old_or_new_walker(Node *node, void *context);
static bool locate_var_of_level_walker(Node *node,
locate_var_of_level_context *context);
static bool pull_var_clause_walker(Node *node,
@@ -492,6 +493,49 @@ contain_vars_of_level_walker(Node *node, int *sublevels_up)
}
+/*
+ * contain_vars_returning_old_or_new
+ * Recursively scan a clause to discover whether it contains any Var nodes
+ * (of the current query level) whose varreturningtype is VAR_RETURNING_OLD
+ * or VAR_RETURNING_NEW.
+ *
+ * Returns true if any found.
+ *
+ * Any ReturningExprs are also detected --- if an OLD/NEW Var was rewritten,
+ * we still regard this as a clause that returns OLD/NEW values.
+ *
+ * Does not examine subqueries, therefore must only be used after reduction
+ * of sublinks to subplans!
+ */
+bool
+contain_vars_returning_old_or_new(Node *node)
+{
+ return contain_vars_returning_old_or_new_walker(node, NULL);
+}
+
+static bool
+contain_vars_returning_old_or_new_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ if (((Var *) node)->varlevelsup == 0 &&
+ ((Var *) node)->varreturningtype != VAR_RETURNING_DEFAULT)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup == 0)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ return expression_tree_walker(node, contain_vars_returning_old_or_new_walker,
+ context);
+}
+
+
/*
* locate_var_of_level
* Find the parse location of any Var of the specified query level.
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index 3864a675d2..dae169f6ee 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -641,8 +641,8 @@ transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -1054,7 +1054,7 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
* contain only the target relation, removing any entries added in a
* sub-SELECT or VALUES list.
*/
- if (stmt->onConflictClause || stmt->returningList)
+ if (stmt->onConflictClause || stmt->returningClause)
{
pstate->p_namespace = NIL;
addNSItemToQuery(pstate, pstate->p_target_nsitem,
@@ -1067,10 +1067,9 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
stmt->onConflictClause);
/* Process RETURNING, if any. */
- if (stmt->returningList)
- qry->returningList = transformReturningList(pstate,
- stmt->returningList,
- EXPR_KIND_RETURNING);
+ if (stmt->returningClause)
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -2548,8 +2547,8 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/*
* Now we are done with SELECT-like processing, and can get on with
@@ -2645,18 +2644,120 @@ transformUpdateTargetList(ParseState *pstate, List *origTlist)
}
/*
- * transformReturningList -
+ * addNSItemForReturning -
+ * add a ParseNamespaceItem for the OLD or NEW alias in RETURNING.
+ */
+static void
+addNSItemForReturning(ParseState *pstate, const char *aliasname,
+ VarReturningType returning_type)
+{
+ List *colnames;
+ int numattrs;
+ ParseNamespaceColumn *nscolumns;
+ ParseNamespaceItem *nsitem;
+
+ /* copy per-column data from the target relation */
+ colnames = pstate->p_target_nsitem->p_rte->eref->colnames;
+ numattrs = list_length(colnames);
+
+ nscolumns = (ParseNamespaceColumn *)
+ palloc(numattrs * sizeof(ParseNamespaceColumn));
+
+ memcpy(nscolumns, pstate->p_target_nsitem->p_nscolumns,
+ numattrs * sizeof(ParseNamespaceColumn));
+
+ /* mark all columns as returning OLD/NEW */
+ for (int i = 0; i < numattrs; i++)
+ nscolumns[i].p_varreturningtype = returning_type;
+
+ /* build the nsitem, copying most fields from the target relation */
+ nsitem = (ParseNamespaceItem *) palloc(sizeof(ParseNamespaceItem));
+ nsitem->p_names = makeAlias(aliasname, colnames);
+ nsitem->p_rte = pstate->p_target_nsitem->p_rte;
+ nsitem->p_rtindex = pstate->p_target_nsitem->p_rtindex;
+ nsitem->p_perminfo = pstate->p_target_nsitem->p_perminfo;
+ nsitem->p_nscolumns = nscolumns;
+ nsitem->p_returning_type = returning_type;
+
+ /* add it to the query namespace as a table-only item */
+ addNSItemToQuery(pstate, nsitem, false, true, false);
+}
+
+/*
+ * transformReturningClause -
* handle a RETURNING clause in INSERT/UPDATE/DELETE/MERGE
*/
-List *
-transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind)
+void
+transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind)
{
- List *rlist;
+ int save_nslen;
int save_next_resno;
- if (returningList == NIL)
- return NIL; /* nothing to do */
+ if (returningClause == NULL)
+ return; /* nothing to do */
+
+ /*
+ * Scan RETURNING WITH(...) options for OLD/NEW alias names. Complain if
+ * there is any conflict with existing relations.
+ */
+ foreach_node(ReturningOption, option, returningClause->options)
+ {
+ switch (option->option)
+ {
+ case RETURNING_OPTION_OLD:
+ if (qry->returningOldAlias != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "OLD"),
+ parser_errposition(pstate, option->location));
+ qry->returningOldAlias = option->value;
+ break;
+
+ case RETURNING_OPTION_NEW:
+ if (qry->returningNewAlias != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "NEW"),
+ parser_errposition(pstate, option->location));
+ qry->returningNewAlias = option->value;
+ break;
+
+ default:
+ elog(ERROR, "unrecognized returning option: %d", option->option);
+ }
+
+ if (refnameNamespaceItem(pstate, NULL, option->value, -1, NULL) != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_ALIAS),
+ errmsg("table name \"%s\" specified more than once",
+ option->value),
+ parser_errposition(pstate, option->location));
+ }
+
+ /*
+ * If no OLD/NEW aliases specified, use "old"/"new" unless masked by
+ * existing relations.
+ */
+ if (qry->returningOldAlias == NULL &&
+ refnameNamespaceItem(pstate, NULL, "old", -1, NULL) == NULL)
+ qry->returningOldAlias = "old";
+ if (qry->returningNewAlias == NULL &&
+ refnameNamespaceItem(pstate, NULL, "new", -1, NULL) == NULL)
+ qry->returningNewAlias = "new";
+
+ /*
+ * Add the OLD and NEW aliases to the query namespace, for use in
+ * expressions in the RETURNING list.
+ */
+ save_nslen = list_length(pstate->p_namespace);
+ if (qry->returningOldAlias != NULL)
+ addNSItemForReturning(pstate, qry->returningOldAlias, VAR_RETURNING_OLD);
+ if (qry->returningNewAlias != NULL)
+ addNSItemForReturning(pstate, qry->returningNewAlias, VAR_RETURNING_NEW);
/*
* We need to assign resnos starting at one in the RETURNING list. Save
@@ -2666,8 +2767,10 @@ transformReturningList(ParseState *pstate, List *returningList,
save_next_resno = pstate->p_next_resno;
pstate->p_next_resno = 1;
- /* transform RETURNING identically to a SELECT targetlist */
- rlist = transformTargetList(pstate, returningList, exprKind);
+ /* transform RETURNING expressions identically to a SELECT targetlist */
+ qry->returningList = transformTargetList(pstate,
+ returningClause->exprs,
+ exprKind);
/*
* Complain if the nonempty tlist expanded to nothing (which is possible
@@ -2675,24 +2778,23 @@ transformReturningList(ParseState *pstate, List *returningList,
* allow this, the parsed Query will look like it didn't have RETURNING,
* with results that would probably surprise the user.
*/
- if (rlist == NIL)
+ if (qry->returningList == NIL)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("RETURNING must have at least one column"),
parser_errposition(pstate,
- exprLocation(linitial(returningList)))));
+ exprLocation(linitial(returningClause->exprs)))));
/* mark column origins */
- markTargetListOrigins(pstate, rlist);
+ markTargetListOrigins(pstate, qry->returningList);
/* resolve any still-unresolved output columns as being type text */
if (pstate->p_resolve_unknowns)
- resolveTargetListUnknowns(pstate, rlist);
+ resolveTargetListUnknowns(pstate, qry->returningList);
/* restore state */
+ pstate->p_namespace = list_truncate(pstate->p_namespace, save_nslen);
pstate->p_next_resno = save_next_resno;
-
- return rlist;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index bd5ebb35c4..d345f110ec 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -265,6 +265,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
MergeWhenClause *mergewhen;
struct KeyActions *keyactions;
struct KeyAction *keyaction;
+ ReturningClause *retclause;
+ ReturningOptionKind retoptionkind;
}
%type <node> stmt toplevel_stmt schema_stmt routine_body_stmt
@@ -434,7 +436,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
opclass_purpose opt_opfamily transaction_mode_list_or_empty
OptTableFuncElementList TableFuncElementList opt_type_modifiers
prep_type_clause
- execute_param_clause using_clause returning_clause
+ execute_param_clause using_clause
+ returning_with_clause returning_options
opt_enum_val_list enum_val_list table_func_column_list
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
@@ -443,6 +446,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
vacuum_relation_list opt_vacuum_relation_list
drop_option_list pub_obj_list
+%type <retclause> returning_clause
+%type <node> returning_option
+%type <retoptionkind> returning_option_kind
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
@@ -12177,7 +12183,7 @@ InsertStmt:
{
$5->relation = $4;
$5->onConflictClause = $6;
- $5->returningList = $7;
+ $5->returningClause = $7;
$5->withClause = $1;
$5->stmt_location = @$;
$$ = (Node *) $5;
@@ -12311,8 +12317,45 @@ opt_conf_expr:
;
returning_clause:
- RETURNING target_list { $$ = $2; }
- | /* EMPTY */ { $$ = NIL; }
+ RETURNING returning_with_clause target_list
+ {
+ ReturningClause *n = makeNode(ReturningClause);
+
+ n->options = $2;
+ n->exprs = $3;
+ $$ = n;
+ }
+ | /* EMPTY */
+ {
+ $$ = NULL;
+ }
+ ;
+
+returning_with_clause:
+ WITH '(' returning_options ')' { $$ = $3; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+returning_options:
+ returning_option { $$ = list_make1($1); }
+ | returning_options ',' returning_option { $$ = lappend($1, $3); }
+ ;
+
+returning_option:
+ returning_option_kind AS ColId
+ {
+ ReturningOption *n = makeNode(ReturningOption);
+
+ n->option = $1;
+ n->value = $3;
+ n->location = @1;
+ $$ = (Node *) n;
+ }
+ ;
+
+returning_option_kind:
+ OLD { $$ = RETURNING_OPTION_OLD; }
+ | NEW { $$ = RETURNING_OPTION_NEW; }
;
@@ -12331,7 +12374,7 @@ DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
n->relation = $4;
n->usingClause = $5;
n->whereClause = $6;
- n->returningList = $7;
+ n->returningClause = $7;
n->withClause = $1;
n->stmt_location = @$;
$$ = (Node *) n;
@@ -12406,7 +12449,7 @@ UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
n->targetList = $5;
n->fromClause = $6;
n->whereClause = $7;
- n->returningList = $8;
+ n->returningClause = $8;
n->withClause = $1;
n->stmt_location = @$;
$$ = (Node *) n;
@@ -12485,7 +12528,7 @@ MergeStmt:
m->sourceRelation = $6;
m->joinCondition = $8;
m->mergeWhenClauses = $9;
- m->returningList = $10;
+ m->returningClause = $10;
m->stmt_location = @$;
$$ = (Node *) m;
diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c
index 979926b605..725591fa5c 100644
--- a/src/backend/parser/parse_clause.c
+++ b/src/backend/parser/parse_clause.c
@@ -1585,6 +1585,7 @@ transformFromClauseItem(ParseState *pstate, Node *n,
jnsitem->p_cols_visible = true;
jnsitem->p_lateral_only = false;
jnsitem->p_lateral_ok = true;
+ jnsitem->p_returning_type = VAR_RETURNING_DEFAULT;
/* Per SQL, we must check for alias conflicts */
checkNameSpaceConflicts(pstate, list_make1(jnsitem), my_namespace);
my_namespace = lappend(my_namespace, jnsitem);
@@ -1647,6 +1648,7 @@ buildVarFromNSColumn(ParseState *pstate, ParseNamespaceColumn *nscol)
nscol->p_varcollid,
0);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index c2806297aa..18769a651b 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -2619,6 +2619,13 @@ transformWholeRowRef(ParseState *pstate, ParseNamespaceItem *nsitem,
* point, there seems no harm in expanding it now rather than during
* planning.
*
+ * Note that if the nsitem is an OLD/NEW alias for the target RTE (as can
+ * appear in a RETURNING list), its alias won't match the target RTE's
+ * alias, but we still want to make a whole-row Var here rather than a
+ * RowExpr, for consistency with direct references to the target RTE, and
+ * so that any dropped columns are handled correctly. Thus we also check
+ * p_returning_type here.
+ *
* Note that if the RTE is a function returning scalar, we create just a
* plain reference to the function value, not a composite containing a
* single column. This is pretty inconsistent at first sight, but it's
@@ -2626,13 +2633,17 @@ transformWholeRowRef(ParseState *pstate, ParseNamespaceItem *nsitem,
* "rel.*" mean the same thing for composite relations, so why not for
* scalar functions...
*/
- if (nsitem->p_names == nsitem->p_rte->eref)
+ if (nsitem->p_names == nsitem->p_rte->eref ||
+ nsitem->p_returning_type != VAR_RETURNING_DEFAULT)
{
Var *result;
result = makeWholeRowVar(nsitem->p_rte, nsitem->p_rtindex,
sublevels_up, true);
+ /* mark Var for RETURNING OLD/NEW, as necessary */
+ result->varreturningtype = nsitem->p_returning_type;
+
/* location is not filled in by makeWholeRowVar */
result->location = location;
@@ -2655,9 +2666,8 @@ transformWholeRowRef(ParseState *pstate, ParseNamespaceItem *nsitem,
* are in the RTE. We needn't worry about marking the RTE for SELECT
* access, as the common columns are surely so marked already.
*/
- expandRTE(nsitem->p_rte, nsitem->p_rtindex,
- sublevels_up, location, false,
- NULL, &fields);
+ expandRTE(nsitem->p_rte, nsitem->p_rtindex, sublevels_up,
+ nsitem->p_returning_type, location, false, NULL, &fields);
rowexpr = makeNode(RowExpr);
rowexpr->args = list_truncate(fields,
list_length(nsitem->p_names->colnames));
diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c
index 87df79027d..0eb8bb411a 100644
--- a/src/backend/parser/parse_merge.c
+++ b/src/backend/parser/parse_merge.c
@@ -247,8 +247,8 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt)
qry->jointree = makeFromExpr(pstate->p_joinlist, NULL);
/* Transform the RETURNING list, if any */
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_MERGE_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_MERGE_RETURNING);
/*
* We now have a good query shape, so now look at the WHEN conditions and
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 8075b1b8a1..610d879b26 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -91,11 +91,13 @@ static void markRTEForSelectPriv(ParseState *pstate,
int rtindex, AttrNumber col);
static void expandRelation(Oid relid, Alias *eref,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static int specialAttNum(const char *attname);
@@ -763,6 +765,9 @@ scanNSItemForColumn(ParseState *pstate, ParseNamespaceItem *nsitem,
}
var->location = location;
+ /* Mark Var for RETURNING OLD/NEW, as necessary */
+ var->varreturningtype = nsitem->p_returning_type;
+
/* Mark Var if it's nulled by any outer joins */
markNullableIfNeeded(pstate, var);
@@ -1336,6 +1341,7 @@ buildNSItemFromTupleDesc(RangeTblEntry *rte, Index rtindex,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -1399,6 +1405,7 @@ buildNSItemFromLists(RangeTblEntry *rte, Index rtindex,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2300,6 +2307,7 @@ addRangeTableEntryForJoin(ParseState *pstate,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2720,9 +2728,10 @@ addNSItemToQuery(ParseState *pstate, ParseNamespaceItem *nsitem,
* results. If include_dropped is true then empty strings and NULL constants
* (not Vars!) are returned for dropped columns.
*
- * rtindex, sublevels_up, and location are the varno, varlevelsup, and location
- * values to use in the created Vars. Ordinarily rtindex should match the
- * actual position of the RTE in its rangetable.
+ * rtindex, sublevels_up, returning_type, and location are the varno,
+ * varlevelsup, varreturningtype, and location values to use in the created
+ * Vars. Ordinarily rtindex should match the actual position of the RTE in
+ * its rangetable.
*
* The output lists go into *colnames and *colvars.
* If only one of the two kinds of output list is needed, pass NULL for the
@@ -2730,6 +2739,7 @@ addNSItemToQuery(ParseState *pstate, ParseNamespaceItem *nsitem,
*/
void
expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -2745,7 +2755,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
case RTE_RELATION:
/* Ordinary relation RTE */
expandRelation(rte->relid, rte->eref,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up, returning_type, location,
include_dropped, colnames, colvars);
break;
case RTE_SUBQUERY:
@@ -2792,6 +2802,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
exprTypmod((Node *) te->expr),
exprCollation((Node *) te->expr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2829,7 +2840,8 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
Assert(tupdesc);
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up,
+ returning_type, location,
include_dropped, colnames, colvars);
}
else if (functypclass == TYPEFUNC_SCALAR)
@@ -2849,6 +2861,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
exprTypmod(rtfunc->funcexpr),
exprCollation(rtfunc->funcexpr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2891,6 +2904,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
attrtypmod,
attrcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
}
@@ -2920,6 +2934,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
InvalidOid,
sublevels_up);
+ varnode->varreturningtype = returning_type;
*colvars = lappend(*colvars, varnode);
}
}
@@ -3002,6 +3017,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
exprTypmod(avar),
exprCollation(avar),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3057,6 +3073,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
varnode = makeVar(rtindex, varattno,
coltype, coltypmod, colcoll,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3089,6 +3106,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
*/
static void
expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3097,7 +3115,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
/* Get the tupledesc and turn it over to expandTupleDesc */
rel = relation_open(relid, AccessShareLock);
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
- rtindex, sublevels_up,
+ rtindex, sublevels_up, returning_type,
location, include_dropped,
colnames, colvars);
relation_close(rel, AccessShareLock);
@@ -3115,6 +3133,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3175,6 +3194,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
attr->atttypid, attr->atttypmod,
attr->attcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3227,6 +3247,7 @@ expandNSItemVars(ParseState *pstate, ParseNamespaceItem *nsitem,
nscol->p_varcollid,
sublevels_up);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
var->location = location;
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
index 76bf88c3ca..f90afe21ad 100644
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -1550,8 +1550,8 @@ expandRecordVariable(ParseState *pstate, Var *var, int levelsup)
*lvar;
int i;
- expandRTE(rte, var->varno, 0, var->location, false,
- &names, &vars);
+ expandRTE(rte, var->varno, 0, var->varreturningtype,
+ var->location, false, &names, &vars);
tupleDesc = CreateTemplateTupleDesc(list_length(vars));
i = 1;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index ab2e2cd647..e84edf89c6 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -641,6 +641,7 @@ rewriteRuleAction(Query *parsetree,
0,
rt_fetch(new_varno, sub_action->rtable),
parsetree->targetList,
+ sub_action->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -674,10 +675,15 @@ rewriteRuleAction(Query *parsetree,
rt_fetch(parsetree->resultRelation,
parsetree->rtable),
rule_action->returningList,
+ rule_action->resultRelation,
REPLACEVARS_REPORT_ERROR,
0,
&rule_action->hasSubLinks);
+ /* use triggering query's aliases for OLD and NEW in RETURNING list */
+ rule_action->returningOldAlias = parsetree->returningOldAlias;
+ rule_action->returningNewAlias = parsetree->returningNewAlias;
+
/*
* There could have been some SubLinks in parsetree's returningList,
* in which case we'd better mark the rule_action correctly.
@@ -2358,6 +2364,7 @@ CopyAndAddInvertedQual(Query *parsetree,
rt_fetch(rt_index,
parsetree->rtable),
parsetree->targetList,
+ parsetree->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -3582,6 +3589,7 @@ rewriteTargetView(Query *parsetree, Relation view)
0,
view_rte,
view_targetlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
NULL);
@@ -3733,6 +3741,7 @@ rewriteTargetView(Query *parsetree, Relation view)
0,
view_rte,
tmp_tlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
&parsetree->hasSubLinks);
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
index f4e687c986..a3640885ed 100644
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -810,6 +810,14 @@ IncrementVarSublevelsUp_walker(Node *node,
phv->phlevelsup += context->delta_sublevels_up;
/* fall through to recurse into argument */
}
+ if (IsA(node, ReturningExpr))
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+
+ if (rexpr->retlevelsup >= context->min_sublevels_up)
+ rexpr->retlevelsup += context->delta_sublevels_up;
+ /* fall through to recurse into argument */
+ }
if (IsA(node, RangeTblEntry))
{
RangeTblEntry *rte = (RangeTblEntry *) node;
@@ -875,6 +883,67 @@ IncrementVarSublevelsUp_rtable(List *rtable, int delta_sublevels_up,
QTW_EXAMINE_RTES_BEFORE);
}
+/*
+ * SetVarReturningType - adjust Var nodes for a specified varreturningtype.
+ *
+ * Find all Var nodes referring to the specified result relation in the given
+ * expression and set their varreturningtype to the specified value.
+ *
+ * NOTE: although this has the form of a walker, we cheat and modify the
+ * Var nodes in-place. The given expression tree should have been copied
+ * earlier to ensure that no unwanted side-effects occur!
+ */
+
+typedef struct
+{
+ int result_relation;
+ int sublevels_up;
+ VarReturningType returning_type;
+} SetVarReturningType_context;
+
+static bool
+SetVarReturningType_walker(Node *node, SetVarReturningType_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ Var *var = (Var *) node;
+
+ if (var->varno == context->result_relation &&
+ var->varlevelsup == context->sublevels_up)
+ var->varreturningtype = context->returning_type;
+
+ return false;
+ }
+
+ if (IsA(node, Query))
+ {
+ /* Recurse into subselects */
+ bool result;
+
+ context->sublevels_up++;
+ result = query_tree_walker((Query *) node, SetVarReturningType_walker,
+ context, 0);
+ context->sublevels_up--;
+ return result;
+ }
+ return expression_tree_walker(node, SetVarReturningType_walker, context);
+}
+
+static void
+SetVarReturningType(Node *node, int result_relation, int sublevels_up,
+ VarReturningType returning_type)
+{
+ SetVarReturningType_context context;
+
+ context.result_relation = result_relation;
+ context.sublevels_up = sublevels_up;
+ context.returning_type = returning_type;
+
+ /* Expect to start with an expression */
+ SetVarReturningType_walker(node, &context);
+}
/*
* rangeTableEntry_used - detect whether an RTE is referenced somewhere
@@ -1640,6 +1709,15 @@ map_variable_attnos(Node *node,
* relation. This is needed to handle whole-row Vars referencing the target.
* We expand such Vars into RowExpr constructs.
*
+ * In addition, for INSERT/UPDATE/DELETE/MERGE queries, the caller must
+ * provide result_relation, the index of the result relation in the rewritten
+ * query. This is needed to handle OLD/NEW RETURNING list Vars referencing
+ * target_varno. When such Vars are expanded, their varreturningtype is
+ * copied onto any replacement Vars referencing result_relation. In addition,
+ * if the replacement expression from the targetlist is not simply a Var
+ * referencing result_relation, it is wrapped in a ReturningExpr node (causing
+ * the executor to return NULL if the OLD/NEW row doesn't exist).
+ *
* outer_hasSubLinks works the same as for replace_rte_variables().
*/
@@ -1647,6 +1725,7 @@ typedef struct
{
RangeTblEntry *target_rte;
List *targetlist;
+ int result_relation;
ReplaceVarsNoMatchOption nomatch_option;
int nomatch_varno;
} ReplaceVarsFromTargetList_context;
@@ -1671,10 +1750,13 @@ ReplaceVarsFromTargetList_callback(Var *var,
* dropped columns. If the var is RECORD (ie, this is a JOIN), then
* omit dropped columns. In the latter case, attach column names to
* the RowExpr for use of the executor and ruleutils.c.
+ *
+ * The varreturningtype is copied onto each individual field Var, so
+ * that it is handled correctly when we recurse.
*/
expandRTE(rcon->target_rte,
- var->varno, var->varlevelsup, var->location,
- (var->vartype != RECORDOID),
+ var->varno, var->varlevelsup, var->varreturningtype,
+ var->location, (var->vartype != RECORDOID),
&colnames, &fields);
/* Adjust the generated per-field Vars... */
fields = (List *) replace_rte_variables_mutator((Node *) fields,
@@ -1686,6 +1768,18 @@ ReplaceVarsFromTargetList_callback(Var *var,
rowexpr->colnames = (var->vartype == RECORDOID) ? colnames : NIL;
rowexpr->location = var->location;
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = var->varreturningtype == VAR_RETURNING_OLD;
+ rexpr->retexpr = (Expr *) rowexpr;
+
+ return (Node *) rexpr;
+ }
+
return (Node *) rowexpr;
}
@@ -1751,6 +1845,34 @@ ReplaceVarsFromTargetList_callback(Var *var,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("NEW variables in ON UPDATE rules cannot reference columns that are part of a multiple assignment in the subject UPDATE command")));
+ /* Handle any OLD/NEW RETURNING list Vars */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ /*
+ * Copy varreturningtype onto any Vars in the tlist item that
+ * refer to result_relation (which had better be non-zero).
+ */
+ if (rcon->result_relation == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+
+ SetVarReturningType((Node *) newnode, rcon->result_relation,
+ var->varlevelsup, var->varreturningtype);
+
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (!IsA(newnode, Var) ||
+ ((Var *) newnode)->varno != rcon->result_relation ||
+ ((Var *) newnode)->varlevelsup != var->varlevelsup)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = (var->varreturningtype == VAR_RETURNING_OLD);
+ rexpr->retexpr = newnode;
+
+ newnode = (Expr *) rexpr;
+ }
+ }
+
return (Node *) newnode;
}
}
@@ -1760,6 +1882,7 @@ ReplaceVarsFromTargetList(Node *node,
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks)
@@ -1768,6 +1891,7 @@ ReplaceVarsFromTargetList(Node *node,
context.target_rte = target_rte;
context.targetlist = targetlist;
+ context.result_relation = result_relation;
context.nomatch_option = nomatch_option;
context.nomatch_varno = nomatch_varno;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index be1f1f50b7..0f396794ac 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -167,6 +167,8 @@ typedef struct
List *subplans; /* List of Plan trees for SubPlans */
List *ctes; /* List of CommonTableExpr nodes */
AppendRelInfo **appendrels; /* Array of AppendRelInfo nodes, or NULL */
+ char *ret_old_alias; /* alias for OLD in RETURNING list */
+ char *ret_new_alias; /* alias for NEW in RETURNING list */
/* Workspace for column alias assignment: */
bool unique_using; /* Are we making USING names globally unique */
List *using_names; /* List of assigned names for USING columns */
@@ -426,6 +428,7 @@ static void get_merge_query_def(Query *query, deparse_context *context);
static void get_utility_query_def(Query *query, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context);
static void get_target_list(List *targetList, deparse_context *context);
+static void get_returning_clause(Query *query, deparse_context *context);
static void get_setop_query(Node *setOp, Query *query,
deparse_context *context);
static Node *get_rule_sortgroupclause(Index ref, List *tlist,
@@ -3800,6 +3803,10 @@ deparse_context_for_plan_tree(PlannedStmt *pstmt, List *rtable_names)
* the most-closely-nested first. This is needed to resolve PARAM_EXEC
* Params. Note we assume that all the Plan nodes share the same rtable.
*
+ * For a ModifyTable plan, we might also need to resolve references to OLD/NEW
+ * variables in the RETURNING list, so we copy the alias names of the OLD and
+ * NEW rows from the ModifyTable plan node.
+ *
* Once this function has been called, deparse_expression() can be called on
* subsidiary expression(s) of the specified Plan node. To deparse
* expressions of a different Plan node in the same Plan tree, re-call this
@@ -3820,6 +3827,13 @@ set_deparse_context_plan(List *dpcontext, Plan *plan, List *ancestors)
dpns->ancestors = ancestors;
set_deparse_plan(dpns, plan);
+ /* For ModifyTable, set aliases for OLD and NEW in RETURNING */
+ if (IsA(plan, ModifyTable))
+ {
+ dpns->ret_old_alias = ((ModifyTable *) plan)->returningOldAlias;
+ dpns->ret_new_alias = ((ModifyTable *) plan)->returningNewAlias;
+ }
+
return dpcontext;
}
@@ -4017,6 +4031,8 @@ set_deparse_for_query(deparse_namespace *dpns, Query *query,
dpns->subplans = NIL;
dpns->ctes = query->cteList;
dpns->appendrels = NULL;
+ dpns->ret_old_alias = query->returningOldAlias;
+ dpns->ret_new_alias = query->returningNewAlias;
/* Assign a unique relation alias to each RTE */
set_rtable_names(dpns, parent_namespaces, NULL);
@@ -4411,8 +4427,8 @@ set_relation_column_names(deparse_namespace *dpns, RangeTblEntry *rte,
if (rte->rtekind == RTE_FUNCTION && rte->functions != NIL)
{
/* Since we're not creating Vars, rtindex etc. don't matter */
- expandRTE(rte, 1, 0, -1, true /* include dropped */ ,
- &colnames, NULL);
+ expandRTE(rte, 1, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , &colnames, NULL);
}
else
colnames = rte->eref->colnames;
@@ -6338,6 +6354,42 @@ get_target_list(List *targetList, deparse_context *context)
pfree(targetbuf.data);
}
+static void
+get_returning_clause(Query *query, deparse_context *context)
+{
+ StringInfo buf = context->buf;
+
+ if (query->returningList)
+ {
+ bool have_with = false;
+
+ appendContextKeyword(context, " RETURNING",
+ -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
+
+ /* Add WITH (OLD/NEW) options, if they're not the defaults */
+ if (query->returningOldAlias && strcmp(query->returningOldAlias, "old") != 0)
+ {
+ appendStringInfo(buf, " WITH (OLD AS %s", query->returningOldAlias);
+ have_with = true;
+ }
+ if (query->returningNewAlias && strcmp(query->returningNewAlias, "new") != 0)
+ {
+ if (have_with)
+ appendStringInfo(buf, ", NEW AS %s", query->returningNewAlias);
+ else
+ {
+ appendStringInfo(buf, " WITH (NEW AS %s", query->returningNewAlias);
+ have_with = true;
+ }
+ }
+ if (have_with)
+ appendStringInfoChar(buf, ')');
+
+ /* Add the returning expressions themselves */
+ get_target_list(query->returningList, context);
+ }
+}
+
static void
get_setop_query(Node *setOp, Query *query, deparse_context *context)
{
@@ -7018,11 +7070,7 @@ get_insert_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7074,11 +7122,7 @@ get_update_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7277,11 +7321,7 @@ get_delete_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7440,11 +7480,7 @@ get_merge_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7592,7 +7628,13 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
}
rte = rt_fetch(varno, dpns->rtable);
- refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+ if (var->varreturningtype == VAR_RETURNING_OLD)
+ refname = dpns->ret_old_alias;
+ else if (var->varreturningtype == VAR_RETURNING_NEW)
+ refname = dpns->ret_new_alias;
+ else
+ refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+
colinfo = deparse_columns_fetch(varno, dpns);
attnum = varattno;
}
@@ -7706,7 +7748,8 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
attname = get_rte_attribute_name(rte, attnum);
}
- need_prefix = (context->varprefix || attname == NULL);
+ need_prefix = (context->varprefix || attname == NULL ||
+ var->varreturningtype != VAR_RETURNING_DEFAULT);
/*
* If we're considering a plain Var in an ORDER BY (but not GROUP BY)
@@ -8757,6 +8800,7 @@ isSimpleNode(Node *node, Node *parentNode, int prettyFlags)
case T_SQLValueFunction:
case T_XmlExpr:
case T_NextValueExpr:
+ case T_ReturningExpr:
case T_NullIfExpr:
case T_Aggref:
case T_GroupingFunc:
@@ -8879,6 +8923,7 @@ isSimpleNode(Node *node, Node *parentNode, int prettyFlags)
case T_CoalesceExpr: /* own parentheses */
case T_MinMaxExpr: /* own parentheses */
case T_XmlExpr: /* own parentheses */
+ case T_ReturningExpr: /* own parentheses */
case T_NullIfExpr: /* other separators */
case T_Aggref: /* own parentheses */
case T_GroupingFunc: /* own parentheses */
@@ -8931,6 +8976,7 @@ isSimpleNode(Node *node, Node *parentNode, int prettyFlags)
case T_CoalesceExpr: /* own parentheses */
case T_MinMaxExpr: /* own parentheses */
case T_XmlExpr: /* own parentheses */
+ case T_ReturningExpr: /* own parentheses */
case T_NullIfExpr: /* other separators */
case T_Aggref: /* own parentheses */
case T_GroupingFunc: /* own parentheses */
@@ -10288,6 +10334,20 @@ get_rule_expr(Node *node, deparse_context *context,
}
break;
+ case T_ReturningExpr:
+ {
+ ReturningExpr *retExpr = (ReturningExpr *) node;
+
+ /*
+ * We cannot see a ReturningExpr in rule deparsing, only while
+ * EXPLAINing a query plan (ReturningExpr nodes are only ever
+ * adding during query rewriting). Just display the expression
+ * returned (an expanded view column).
+ */
+ get_rule_expr((Node *) retExpr->retexpr, context, showimplicit);
+ }
+ break;
+
case T_PartitionBoundSpec:
{
PartitionBoundSpec *spec = (PartitionBoundSpec *) node;
diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h
index 56fb0d0adb..6fd675c54a 100644
--- a/src/include/executor/execExpr.h
+++ b/src/include/executor/execExpr.h
@@ -26,9 +26,9 @@ struct JsonConstructorExprState;
/* Bits in ExprState->flags (see also execnodes.h for public flag bits): */
/* expression's interpreter has been initialized */
-#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 1)
+#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 5)
/* jump-threading is in use */
-#define EEO_FLAG_DIRECT_THREADED (1 << 2)
+#define EEO_FLAG_DIRECT_THREADED (1 << 6)
/* Typical API for out-of-line evaluation subroutines */
typedef void (*ExecEvalSubroutine) (ExprState *state,
@@ -72,16 +72,22 @@ typedef enum ExprEvalOp
EEOP_INNER_FETCHSOME,
EEOP_OUTER_FETCHSOME,
EEOP_SCAN_FETCHSOME,
+ EEOP_OLD_FETCHSOME,
+ EEOP_NEW_FETCHSOME,
/* compute non-system Var value */
EEOP_INNER_VAR,
EEOP_OUTER_VAR,
EEOP_SCAN_VAR,
+ EEOP_OLD_VAR,
+ EEOP_NEW_VAR,
/* compute system Var value */
EEOP_INNER_SYSVAR,
EEOP_OUTER_SYSVAR,
EEOP_SCAN_SYSVAR,
+ EEOP_OLD_SYSVAR,
+ EEOP_NEW_SYSVAR,
/* compute wholerow Var */
EEOP_WHOLEROW,
@@ -94,6 +100,8 @@ typedef enum ExprEvalOp
EEOP_ASSIGN_INNER_VAR,
EEOP_ASSIGN_OUTER_VAR,
EEOP_ASSIGN_SCAN_VAR,
+ EEOP_ASSIGN_OLD_VAR,
+ EEOP_ASSIGN_NEW_VAR,
/* assign ExprState's resvalue/resnull to a column of its resultslot */
EEOP_ASSIGN_TMP,
@@ -178,6 +186,7 @@ typedef enum ExprEvalOp
EEOP_SQLVALUEFUNCTION,
EEOP_CURRENTOFEXPR,
EEOP_NEXTVALUEEXPR,
+ EEOP_RETURNINGEXPR,
EEOP_ARRAYEXPR,
EEOP_ARRAYCOERCE,
EEOP_ROW,
@@ -301,7 +310,7 @@ typedef struct ExprEvalStep
*/
union
{
- /* for EEOP_INNER/OUTER/SCAN_FETCHSOME */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_FETCHSOME */
struct
{
/* attribute number up to which to fetch (inclusive) */
@@ -314,13 +323,14 @@ typedef struct ExprEvalStep
const TupleTableSlotOps *kind;
} fetch;
- /* for EEOP_INNER/OUTER/SCAN_[SYS]VAR[_FIRST] */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_[SYS]VAR */
struct
{
/* attnum is attr number - 1 for regular VAR ... */
/* but it's just the normal (negative) attr number for SYSVAR */
int attnum;
Oid vartype; /* type OID of variable */
+ VarReturningType varreturningtype; /* return old/new/default */
} var;
/* for EEOP_WHOLEROW */
@@ -349,6 +359,13 @@ typedef struct ExprEvalStep
int resultnum;
} assign_tmp;
+ /* for EEOP_RETURNINGEXPR */
+ struct
+ {
+ uint8 nullflag; /* flag to test if OLD/NEW row is NULL */
+ int jumpdone; /* jump here if OLD/NEW row is NULL */
+ } returningexpr;
+
/* for EEOP_CONST */
struct
{
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 1c7fae0930..bdfb39b7f1 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -629,6 +629,7 @@ extern int ExecCleanTargetListLength(List *targetlist);
extern TupleTableSlot *ExecGetTriggerOldSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetTriggerNewSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetReturningSlot(EState *estate, ResultRelInfo *relInfo);
+extern TupleTableSlot *ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleConversionMap *ExecGetChildToRootMap(ResultRelInfo *resultRelInfo);
extern TupleConversionMap *ExecGetRootToChildMap(ResultRelInfo *resultRelInfo, EState *estate);
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 1590b64392..9535bcb8b4 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -74,11 +74,20 @@ typedef Datum (*ExprStateEvalFunc) (struct ExprState *expression,
/* Bits in ExprState->flags (see also execExpr.h for private flag bits): */
/* expression is for use with ExecQual() */
#define EEO_FLAG_IS_QUAL (1 << 0)
+/* expression refers to OLD table columns */
+#define EEO_FLAG_HAS_OLD (1 << 1)
+/* expression refers to NEW table columns */
+#define EEO_FLAG_HAS_NEW (1 << 2)
+/* OLD table row is NULL in RETURNING list */
+#define EEO_FLAG_OLD_IS_NULL (1 << 3)
+/* NEW table row is NULL in RETURNING list */
+#define EEO_FLAG_NEW_IS_NULL (1 << 4)
typedef struct ExprState
{
NodeTag type;
+#define FIELDNO_EXPRSTATE_FLAGS 1
uint8 flags; /* bitmask of EEO_FLAG_* bits, see above */
/*
@@ -290,6 +299,12 @@ typedef struct ExprContext
#define FIELDNO_EXPRCONTEXT_DOMAINNULL 13
bool domainValue_isNull;
+ /* Tuples that OLD/NEW Var nodes in RETURNING may refer to */
+#define FIELDNO_EXPRCONTEXT_OLDTUPLE 14
+ TupleTableSlot *ecxt_oldtuple;
+#define FIELDNO_EXPRCONTEXT_NEWTUPLE 15
+ TupleTableSlot *ecxt_newtuple;
+
/* Link to containing EState (NULL if a standalone ExprContext) */
struct EState *ecxt_estate;
@@ -504,6 +519,7 @@ typedef struct ResultRelInfo
TupleTableSlot *ri_ReturningSlot; /* for trigger output tuples */
TupleTableSlot *ri_TrigOldSlot; /* for a trigger's old tuple */
TupleTableSlot *ri_TrigNewSlot; /* for a trigger's new tuple */
+ TupleTableSlot *ri_AllNullSlot; /* for RETURNING OLD/NEW */
/* FDW callback functions, if foreign table */
struct FdwRoutine *ri_FdwRoutine;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 0f9462493e..1984c9a3c1 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -197,6 +197,15 @@ typedef struct Query
OnConflictExpr *onConflict; /* ON CONFLICT DO [NOTHING | UPDATE] */
+ /*
+ * The following three fields describe the contents of the RETURNING list
+ * for INSERT/UPDATE/DELETE/MERGE. returningOldAlias and returningNewAlias
+ * are the alias names for OLD and NEW, which may be user-supplied values,
+ * the defaults "old" and "new", or NULL (if the default "old"/"new" is
+ * already in use as the alias for some other relation).
+ */
+ char *returningOldAlias pg_node_attr(query_jumble_ignore);
+ char *returningNewAlias pg_node_attr(query_jumble_ignore);
List *returningList; /* return-values list (of TargetEntry) */
List *groupClause; /* a list of SortGroupClause's */
@@ -1726,6 +1735,41 @@ typedef struct MergeWhenClause
List *values; /* VALUES to INSERT, or NULL */
} MergeWhenClause;
+/*
+ * ReturningOptionKind -
+ * Possible kinds of option in RETURNING WITH(...) list
+ *
+ * Currently, this is used only for specifying OLD/NEW aliases.
+ */
+typedef enum ReturningOptionKind
+{
+ RETURNING_OPTION_OLD, /* specify alias for OLD in RETURNING */
+ RETURNING_OPTION_NEW, /* specify alias for NEW in RETURNING */
+} ReturningOptionKind;
+
+/*
+ * ReturningOption -
+ * An individual option in the RETURNING WITH(...) list
+ */
+typedef struct ReturningOption
+{
+ NodeTag type;
+ ReturningOptionKind option; /* specified option */
+ char *value; /* option's value */
+ ParseLoc location; /* token location, or -1 if unknown */
+} ReturningOption;
+
+/*
+ * ReturningClause -
+ * List of RETURNING expressions, together with any WITH(...) options
+ */
+typedef struct ReturningClause
+{
+ NodeTag type;
+ List *options; /* list of ReturningOption elements */
+ List *exprs; /* list of expressions to return */
+} ReturningClause;
+
/*
* TriggerTransition -
* representation of transition row or table naming clause
@@ -2043,7 +2087,7 @@ typedef struct InsertStmt
List *cols; /* optional: names of the target columns */
Node *selectStmt; /* the source SELECT/VALUES, or NULL */
OnConflictClause *onConflictClause; /* ON CONFLICT clause */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
OverridingKind override; /* OVERRIDING clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
@@ -2060,7 +2104,7 @@ typedef struct DeleteStmt
RangeVar *relation; /* relation to delete from */
List *usingClause; /* optional using clause for more tables */
Node *whereClause; /* qualifications */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
@@ -2077,7 +2121,7 @@ typedef struct UpdateStmt
List *targetList; /* the target list (of ResTarget) */
Node *whereClause; /* qualifications */
List *fromClause; /* optional from clause for more tables */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
@@ -2094,7 +2138,7 @@ typedef struct MergeStmt
Node *sourceRelation; /* source relation */
Node *joinCondition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 4633121689..f2f684be1a 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -238,6 +238,8 @@ typedef struct ModifyTable
List *resultRelations; /* integer list of RT indexes */
List *updateColnosLists; /* per-target-table update_colnos lists */
List *withCheckOptionLists; /* per-target-table WCO lists */
+ char *returningOldAlias; /* alias for OLD in RETURNING lists */
+ char *returningNewAlias; /* alias for NEW in RETURNING lists */
List *returningLists; /* per-target-table RETURNING tlists */
List *fdwPrivLists; /* per-target-table FDW private data lists */
Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index b0ef1952e8..2d98b6f4dc 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -223,6 +223,11 @@ typedef struct Expr
* Note that it affects the meaning of all of varno, varnullingrels, and
* varnosyn, all of which refer to the range table of that query level.
*
+ * varreturningtype is used for Vars that refer to the target relation in the
+ * RETURNING list of data-modifying queries. The default behavior is to
+ * return old values for DELETE and new values for INSERT and UPDATE, but it
+ * is also possible to explicitly request old or new values.
+ *
* In the parser, varnosyn and varattnosyn are either identical to
* varno/varattno, or they specify the column's position in an aliased JOIN
* RTE that hides the semantic referent RTE's refname. This is a syntactic
@@ -244,6 +249,14 @@ typedef struct Expr
#define PRS2_OLD_VARNO 1
#define PRS2_NEW_VARNO 2
+/* Returning behavior for Vars in RETURNING list */
+typedef enum VarReturningType
+{
+ VAR_RETURNING_DEFAULT, /* return OLD for DELETE, else return NEW */
+ VAR_RETURNING_OLD, /* return OLD for DELETE/UPDATE, else NULL */
+ VAR_RETURNING_NEW, /* return NEW for INSERT/UPDATE, else NULL */
+} VarReturningType;
+
typedef struct Var
{
Expr xpr;
@@ -279,6 +292,9 @@ typedef struct Var
*/
Index varlevelsup;
+ /* returning type of this var (see above) */
+ VarReturningType varreturningtype;
+
/*
* varnosyn/varattnosyn are ignored for equality, because Vars with
* different syntactic identifiers are semantically the same as long as
@@ -2128,6 +2144,30 @@ typedef struct InferenceElem
Oid inferopclass; /* OID of att opclass, or InvalidOid */
} InferenceElem;
+/*
+ * ReturningExpr - return OLD/NEW.(expression) in RETURNING list
+ *
+ * This is used when updating an auto-updatable view and returning a view
+ * column that is not simply a Var referring to the base relation. In such
+ * cases, OLD/NEW.viewcol can expand to an arbitrary expression, but the
+ * result is required to be NULL if the OLD/NEW row doesn't exist. To handle
+ * this, the rewriter wraps the expanded expression in a ReturningExpr, which
+ * is equivalent to "CASE WHEN (OLD/NEW row exists) THEN (expr) ELSE NULL".
+ *
+ * A similar situation can arise when rewriting the RETURNING clause of a
+ * rule, which may also contain arbitrary expressions.
+ *
+ * ReturningExpr nodes never appear in a parsed Query --- they are only ever
+ * inserted by the rewriter.
+ */
+typedef struct ReturningExpr
+{
+ Expr xpr;
+ int retlevelsup; /* > 0 if it belongs to outer query */
+ bool retold; /* true for OLD, false for NEW */
+ Expr *retexpr; /* expression to be returned */
+} ReturningExpr;
+
/*--------------------
* TargetEntry -
* a target entry (used in query target lists)
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
index 2e123e08b7..565cfd5800 100644
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -199,6 +199,7 @@ extern void pull_varattnos(Node *node, Index varno, Bitmapset **varattnos);
extern List *pull_vars_of_level(Node *node, int levelsup);
extern bool contain_var_clause(Node *node);
extern bool contain_vars_of_level(Node *node, int levelsup);
+extern bool contain_vars_returning_old_or_new(Node *node);
extern int locate_var_of_level(Node *node, int levelsup);
extern List *pull_var_clause(Node *node, int flags);
extern Node *flatten_join_alias_vars(PlannerInfo *root, Query *query, Node *node);
diff --git a/src/include/optimizer/paramassign.h b/src/include/optimizer/paramassign.h
index 4026b74fab..89d2d0721f 100644
--- a/src/include/optimizer/paramassign.h
+++ b/src/include/optimizer/paramassign.h
@@ -22,6 +22,8 @@ extern Param *replace_outer_agg(PlannerInfo *root, Aggref *agg);
extern Param *replace_outer_grouping(PlannerInfo *root, GroupingFunc *grp);
extern Param *replace_outer_merge_support(PlannerInfo *root,
MergeSupportFunc *msf);
+extern Param *replace_outer_returning(PlannerInfo *root,
+ ReturningExpr *rexpr);
extern Param *replace_nestloop_param_var(PlannerInfo *root, Var *var);
extern Param *replace_nestloop_param_placeholdervar(PlannerInfo *root,
PlaceHolderVar *phv);
diff --git a/src/include/parser/analyze.h b/src/include/parser/analyze.h
index 28b66fccb4..37f3bd32bb 100644
--- a/src/include/parser/analyze.h
+++ b/src/include/parser/analyze.h
@@ -44,8 +44,9 @@ extern List *transformInsertRow(ParseState *pstate, List *exprlist,
bool strip_indirection);
extern List *transformUpdateTargetList(ParseState *pstate,
List *origTlist);
-extern List *transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind);
+extern void transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind);
extern Query *transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree);
extern Query *transformStmt(ParseState *pstate, Node *parseTree);
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index 2375e95c10..ac1a7f55a6 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -295,6 +295,11 @@ struct ParseState
* of SQL:2008 requires us to do it this way. We also use p_lateral_ok to
* forbid LATERAL references to an UPDATE/DELETE target table.
*
+ * While processing the RETURNING clause, special namespace items are added to
+ * refer to the OLD and NEW state of the result relation. These namespace
+ * items have p_returning_type set appropriately, for use when creating Vars.
+ * For convenience, this information is duplicated on each namespace column.
+ *
* At no time should a namespace list contain two entries that conflict
* according to the rules in checkNameSpaceConflicts; but note that those
* are more complicated than "must have different alias names", so in practice
@@ -312,6 +317,7 @@ struct ParseNamespaceItem
bool p_cols_visible; /* Column names visible as unqualified refs? */
bool p_lateral_only; /* Is only visible to LATERAL expressions? */
bool p_lateral_ok; /* If so, does join type allow use? */
+ VarReturningType p_returning_type; /* Is OLD/NEW for use in RETURNING? */
};
/*
@@ -342,6 +348,7 @@ struct ParseNamespaceColumn
Oid p_vartype; /* pg_type OID */
int32 p_vartypmod; /* type modifier value */
Oid p_varcollid; /* OID of collation, or InvalidOid */
+ VarReturningType p_varreturningtype; /* for RETURNING OLD/NEW */
Index p_varnosyn; /* rangetable index of syntactic referent */
AttrNumber p_varattnosyn; /* attribute number of syntactic referent */
bool p_dontexpand; /* not included in star expansion */
diff --git a/src/include/parser/parse_relation.h b/src/include/parser/parse_relation.h
index 91fd8e243b..3dcc1abd2d 100644
--- a/src/include/parser/parse_relation.h
+++ b/src/include/parser/parse_relation.h
@@ -114,6 +114,7 @@ extern void errorMissingRTE(ParseState *pstate, RangeVar *relation) pg_attribute
extern void errorMissingColumn(ParseState *pstate,
const char *relname, const char *colname, int location) pg_attribute_noreturn();
extern void expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
extern List *expandNSItemVars(ParseState *pstate, ParseNamespaceItem *nsitem,
diff --git a/src/include/rewrite/rewriteManip.h b/src/include/rewrite/rewriteManip.h
index ac6d2049e8..15839ac242 100644
--- a/src/include/rewrite/rewriteManip.h
+++ b/src/include/rewrite/rewriteManip.h
@@ -89,6 +89,7 @@ extern Node *ReplaceVarsFromTargetList(Node *node,
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks);
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
index 86943ae253..b5431e1654 100644
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -105,8 +105,8 @@ my %replace_line = (
'SHOW TRANSACTION ISOLATION LEVEL ecpg_into',
'VariableShowStmt SHOW SESSION AUTHORIZATION' =>
'SHOW SESSION AUTHORIZATION ecpg_into',
- 'returning_clause RETURNING target_list' =>
- 'RETURNING target_list opt_ecpg_into',
+ 'returning_clause RETURNING returning_with_clause target_list' =>
+ 'RETURNING returning_with_clause target_list opt_ecpg_into',
'ExecuteStmt EXECUTE name execute_param_clause' =>
'EXECUTE prepared_name execute_param_clause execute_rest',
'ExecuteStmt CREATE OptTemp TABLE create_as_target AS EXECUTE name execute_param_clause opt_with_data'
diff --git a/src/test/isolation/expected/merge-update.out b/src/test/isolation/expected/merge-update.out
index 3063c0c6ab..677263d1ec 100644
--- a/src/test/isolation/expected/merge-update.out
+++ b/src/test/isolation/expected/merge-update.out
@@ -40,12 +40,12 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -98,14 +98,14 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -137,13 +137,13 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step a1: ABORT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+-------------------------
-UPDATE | 2|setup1 updated by merge2a
+merge_action|old |new |key|val
+------------+----------+-------------------------------+---+-------------------------
+UPDATE |(1,setup1)|(2,"setup1 updated by merge2a")| 2|setup1 updated by merge2a
(1 row)
step select2: SELECT * FROM target;
@@ -234,14 +234,14 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
-merge_action|key|val
-------------+---+--------------------------------------------------
-UPDATE | 2|initial updated by pa_merge1 updated by pa_merge2a
-UPDATE | 3|initial source not matched by pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+--------------------------------------------------------+---+--------------------------------------------------
+UPDATE |(1,"initial updated by pa_merge1")|(2,"initial updated by pa_merge1 updated by pa_merge2a")| 2|initial updated by pa_merge1 updated by pa_merge2a
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
(2 rows)
step pa_select2: SELECT * FROM pa_target;
@@ -273,7 +273,7 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
@@ -303,13 +303,13 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+-------------------------------------------------------------
-UPDATE | 3|initial source not matched by pa_merge2a
-UPDATE | 3|initial updated by pa_merge2 source not matched by pa_merge2a
-INSERT | 1|pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+-------------------------------------------------------------------+---+-------------------------------------------------------------
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
+UPDATE |(2,"initial updated by pa_merge2")|(3,"initial updated by pa_merge2 source not matched by pa_merge2a")| 3|initial updated by pa_merge2 source not matched by pa_merge2a
+INSERT | |(1,pa_merge2a) | 1|pa_merge2a
(3 rows)
step pa_select2: SELECT * FROM pa_target;
diff --git a/src/test/isolation/specs/merge-update.spec b/src/test/isolation/specs/merge-update.spec
index a33dcdba53..c718ff646b 100644
--- a/src/test/isolation/specs/merge-update.spec
+++ b/src/test/isolation/specs/merge-update.spec
@@ -95,7 +95,7 @@ step "merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
step "merge2b"
{
@@ -128,7 +128,7 @@ step "pa_merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
# MERGE proceeds only if 'val' unchanged
step "pa_merge2b_when"
diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out
index 28d8551063..05314ad439 100644
--- a/src/test/regress/expected/merge.out
+++ b/src/test/regress/expected/merge.out
@@ -297,13 +297,13 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
- merge_action | tid | balance
---------------+-----+---------
- DELETE | 1 | 10
- DELETE | 2 | 20
- DELETE | 3 | 30
- INSERT | 4 | 40
+RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ DELETE | (1,10) | | 1 | 10
+ DELETE | (2,20) | | 2 | 20
+ DELETE | (3,30) | | 3 | 30
+ INSERT | | (4,40) | 4 | 40
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -994,7 +994,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 THEN
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
NOTICE: BEFORE INSERT STATEMENT trigger
NOTICE: BEFORE UPDATE STATEMENT trigger
NOTICE: BEFORE DELETE STATEMENT trigger
@@ -1009,12 +1009,12 @@ NOTICE: AFTER UPDATE ROW trigger row: (1,10) -> (1,0)
NOTICE: AFTER DELETE STATEMENT trigger
NOTICE: AFTER UPDATE STATEMENT trigger
NOTICE: AFTER INSERT STATEMENT trigger
- merge_action | tid | balance
---------------+-----+---------
- UPDATE | 3 | 10
- INSERT | 4 | 40
- DELETE | 2 | 20
- UPDATE | 1 | 0
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ UPDATE | (3,30) | (3,10) | 3 | 10
+ INSERT | | (4,40) | 4 | 40
+ DELETE | (2,20) | | 2 | 20
+ UPDATE | (1,10) | (1,0) | 1 | 0
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -1436,17 +1436,19 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
WHEN 'DELETE' THEN 'Removed '||t
END AS description;
- action | tid | balance | description
---------+-----+---------+---------------------
- del | 1 | 100 | Removed (1,100)
- upd | 2 | 220 | Added 20 to balance
- ins | 4 | 40 | Inserted (4,40)
+ action | old_tid | old_balance | new_tid | new_balance | delta_balance | tid | balance | description
+--------+---------+-------------+---------+-------------+---------------+-----+---------+---------------------
+ del | 1 | 100 | | | | 1 | 100 | Removed (1,100)
+ upd | 2 | 200 | 2 | 220 | 20 | 2 | 220 | Added 20 to balance
+ ins | | | 4 | 40 | | 4 | 40 | Inserted (4,40)
(3 rows)
ROLLBACK;
@@ -1473,7 +1475,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -1487,14 +1489,14 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
- action | log_action | tid | last_change
---------+------------+-----+---------------------
- DELETE | UPDATE | 1 | Removed (1,100)
- UPDATE | INSERT | 2 | Added 20 to balance
- INSERT | INSERT | 4 | Inserted (4,40)
+ action | old_data | new_data | tid | balance | description | log_action | old_log | new_log | tid | last_change
+--------+----------+----------+-----+---------+---------------------+------------+----------------------+---------------------------+-----+---------------------
+ DELETE | (1,100) | | 1 | 100 | Removed (1,100) | UPDATE | (1,"Original value") | (1,"Removed (1,100)") | 1 | Removed (1,100)
+ UPDATE | (2,200) | (2,220) | 2 | 220 | Added 20 to balance | INSERT | | (2,"Added 20 to balance") | 2 | Added 20 to balance
+ INSERT | | (4,40) | 4 | 40 | Inserted (4,40) | INSERT | | (4,"Inserted (4,40)") | 4 | Inserted (4,40)
(3 rows)
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -1518,11 +1520,11 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
-DELETE 1 100
-UPDATE 2 220
-INSERT 4 40
+DELETE 1 100 \N \N
+UPDATE 2 200 2 220
+INSERT \N \N 4 40
ROLLBACK;
-- SQL function with MERGE ... RETURNING
BEGIN;
@@ -2039,10 +2041,10 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
- merge_action | tid | balance | val
---------------+-----+---------+--------------------------
- UPDATE | 2 | 110 | initial updated by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance | val
+--------------+-----------------+------------------------------------+-----+---------+--------------------------
+ UPDATE | (1,100,initial) | (2,110,"initial updated by merge") | 2 | 110 | initial updated by merge
(1 row)
SELECT * FROM pa_target ORDER BY tid;
@@ -2324,18 +2326,18 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
- merge_action | logts | tid | balance | val
---------------+--------------------------+-----+---------+--------------------------
- UPDATE | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | logts | tid | balance | val
+--------------+--------------------------------------------+---------------------------------------------------------------+--------------------------+-----+---------+--------------------------
+ UPDATE | ("Tue Jan 31 00:00:00 2017",1,100,initial) | ("Tue Jan 31 00:00:00 2017",1,110,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",2,200,initial) | ("Tue Feb 28 00:00:00 2017",2,220,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",3,30,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",4,400,initial) | ("Tue Jan 31 00:00:00 2017",4,440,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",5,500,initial) | ("Tue Feb 28 00:00:00 2017",5,550,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",6,60,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",7,700,initial) | ("Tue Jan 31 00:00:00 2017",7,770,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",8,800,initial) | ("Tue Feb 28 00:00:00 2017",8,880,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",9,90,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
(9 rows)
SELECT * FROM pa_target ORDER BY tid;
diff --git a/src/test/regress/expected/returning.out b/src/test/regress/expected/returning.out
index cb51bb8687..28814ded71 100644
--- a/src/test/regress/expected/returning.out
+++ b/src/test/regress/expected/returning.out
@@ -355,3 +355,543 @@ INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.f3; -- ok
42
(1 row)
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+ERROR: syntax error at or near "nonsuch"
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS so...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+ERROR: table name "foo" specified more than once
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+ERROR: OLD cannot be specified multiple times
+LINE 1: ...EFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) ...
+ ^
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ -> Result
+ Output: 4, NULL::text, 42, '99'::bigint
+(4 rows)
+
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+------+----+----+----+----+----------+-------+----+----+----+----+----+----+----+----
+ | | | | | | foo | (0,4) | 4 | | 42 | 99 | 4 | | 42 | 99
+(1 row)
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (o.tableoid)::regclass, o.ctid, o.f1, o.f2, o.f3, o.f4, (n.tableoid)::regclass, n.ctid, n.f1, n.f2, n.f3, n.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ Conflict Resolution: UPDATE
+ Conflict Arbiter Indexes: foo_f1_idx
+ -> Values Scan on "*VALUES*"
+ Output: "*VALUES*".column1, "*VALUES*".column2, 42, '99'::bigint
+(6 rows)
+
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+----+----------+-------+----+------------+----+----+----+------------+----+----
+ foo | (0,4) | 4 | | 42 | 99 | foo | (0,5) | 4 | conflicted | -1 | 99 | 4 | conflicted | -1 | 99
+ | | | | | | foo | (0,6) | 5 | ok | 42 | 99 | 5 | ok | 42 | 99
+(2 rows)
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ QUERY PLAN
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, old.*, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, new.*, (((old.f4)::text || '->'::text) || (new.f4)::text)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(8 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ tableoid | ctid | f1 | f2 | f3 | f4 | old | tableoid | ctid | f1 | f2 | f3 | f4 | new | change
+----------+-------+----+----+----+----+--------------+----------+-------+----+----+----+-----+---------------+---------
+ foo | (0,6) | 5 | ok | 42 | 99 | (5,ok,42,99) | foo | (0,7) | 5 | ok | 42 | 100 | (5,ok,42,100) | 99->100
+(1 row)
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(6 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+-----+----------+------+----+----+----+----+----+----+----+-----
+ foo | (0,7) | 5 | ok | 42 | 100 | | | | | | | 5 | ok | 42 | 100
+(1 row)
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ -> Result
+ Output: 5, 'subquery test'::text, 42, '99'::bigint
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(16 rows)
+
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ | 109
+(1 row)
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2), (SubPlan 3)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Result
+ Output: (old.f4 = new.f4)
+ SubPlan 2
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 3
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(23 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ ?column? | old_max | new_max
+----------+---------+---------
+ f | 109 | 110
+(1 row)
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(18 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ 110 |
+(1 row)
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, new.f1, new.f2, new.f3, new.f4, foo_2.f1, foo_2.f2, foo_2.f3, foo_2.f4
+ Update on pg_temp.foo foo_2
+ -> Nested Loop
+ Output: (foo_2.f2 || ' (deleted)'::text), '-1'::integer, '-1'::bigint, foo_1.ctid, foo_1.tableoid, foo_2.tableoid, foo_2.ctid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f2, foo_2.f1, foo_2.tableoid, foo_2.ctid
+ Filter: (foo_2.f1 = 4)
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.ctid, foo_1.f1, foo_1.tableoid
+ Filter: (foo_1.f1 = 4)
+(11 rows)
+
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----+------------+----+----+----+----------------------+----+----+----+----------------------+----+----
+ 4 | conflicted | -1 | 99 | 4 | conflicted (deleted) | -1 | -1 | 4 | conflicted (deleted) | -1 | -1
+(1 row)
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, joinme.other, new.f1, new.f2, new.f3, new.f4, joinme.other, foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4, joinme.other, (new.f3 - old.f3)
+ Update on pg_temp.foo foo_1
+ -> Hash Join
+ Output: foo_2.f1, (foo_2.f3 + 1), joinme.ctid, foo_2.ctid, joinme_1.ctid, joinme.other, foo_1.tableoid, foo_1.ctid, foo_2.tableoid
+ Hash Cond: (foo_1.f2 = joinme.f2j)
+ -> Hash Join
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme_1.ctid, joinme_1.f2j
+ Hash Cond: (joinme_1.f2j = foo_1.f2)
+ -> Seq Scan on pg_temp.joinme joinme_1
+ Output: joinme_1.ctid, joinme_1.f2j
+ -> Hash
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Hash
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Hash Join
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Hash Cond: (joinme.f2j = foo_2.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.ctid, joinme.other, joinme.f2j
+ -> Hash
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Filter: (foo_2.f3 = 57)
+(27 rows)
+
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 57 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 1
+(1 row)
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ERROR: column reference "new.f1" is ambiguous
+LINE 3: RETURNING new.f1, new.f4
+ ^
+DETAIL: It could refer to either a PL/pgSQL variable or a table column.
+QUERY: UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4
+CONTEXT: PL/pgSQL function joinview_upd_trig_fn() line 4 at SQL statement
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.joinview
+ Output: old.f1, old.f2, old.f3, old.f4, old.other, new.f1, new.f2, new.f3, new.f4, new.other, joinview.f1, joinview.f2, joinview.f3, joinview.f4, joinview.other, (new.f3 - old.f3)
+ -> Hash Join
+ Output: (foo.f3 + 1), '7'::bigint, ROW(foo.f1, foo.f2, foo.f3, foo.f4, joinme.other), foo.ctid, joinme.ctid, foo.tableoid
+ Hash Cond: (joinme.f2j = foo.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.other, joinme.ctid, joinme.f2j
+ -> Hash
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ -> Seq Scan on pg_temp.foo
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ Filter: (foo.f3 = 58)
+(12 rows)
+
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 1
+(1 row)
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+NOTICE: drop cascades to 3 other objects
+DETAIL: drop cascades to rule voo_i on view voo
+drop cascades to view joinview
+drop cascades to rule foo_del_rule on table foo
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+ERROR: column old.f3 does not exist
+LINE 1: UPDATE foo SET f4 = f4 + 1 RETURNING old.f3;
+ ^
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+ old | new
+-------------------------------+------------------------------
+ (1,xxx,20) | (1,xxx,21)
+ (2,more,141) | (2,more,142)
+ (4,"conflicted (deleted)",-1) | (4,"conflicted (deleted)",0)
+ (3,zoo2,70) | (3,zoo2,71)
+(4 rows)
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+------+----------+-------+-------
+ | | zerocol | (0,1) | (0,1)
+(1 row)
+
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+-------+----------+------+-------
+ zerocol | (0,1) | | | (0,1)
+(1 row)
+
+DROP TABLE zerocol;
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+----------+------+---+---+---+-------------+-------+---+------+----+---+------+----
+ | | | | | foo_part_s1 | (0,1) | 1 | 17.1 | P1 | 1 | 17.1 | P1
+ | | | | | foo_part_s2 | (0,1) | 2 | 17.2 | P2 | 2 | 17.2 | P2
+ | | | | | foo_part_d1 | (0,1) | 3 | 17.3 | P3 | 3 | 17.3 | P3
+ | | | | | foo_part_d2 | (0,1) | 4 | 17.4 | P4 | 4 | 17.4 | P4
+(4 rows)
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_s1 | (0,1) | 1 | 17.1 | P1 | foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | 2 | 18.1 | P1->P2
+(1 row)
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_d1 | (0,1) | 3 | 17.3 | P3 | foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | 1 | 18.3 | P3->P1
+(1 row)
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+--------+-------------+-------+---+------+------------+---+------+------------
+ foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | 3 | 19.3 | P3->P1->P3
+(1 row)
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+------------+-------------+-------+---+------+----------------+---+------+----------------
+ foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | 4 | 20.3 | P3->P1->P3->P4
+(1 row)
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----------------+----------+------+---+---+---+---+------+----------------
+ foo_part_s2 | (0,1) | 2 | 17.2 | P2 | | | | | | 2 | 17.2 | P2
+ foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | | | | | | 2 | 18.1 | P1->P2
+ foo_part_d2 | (0,1) | 4 | 17.4 | P4 | | | | | | 4 | 17.4 | P4
+ foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | | | | | | 4 | 20.3 | P3->P1->P3->P4
+(4 rows)
+
+DROP TABLE foo_parted;
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ WITH u1 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING old.*, new.*
+ ), u2 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (OLD AS o) o.*, new.*
+ ), u3 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (NEW AS n) old.*, n.*
+ )
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.*, n.*, o, n, o.f1 = n.f1, o = n,
+ (SELECT o.f2 = n.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = n.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = n);
+END;
+\sf foo_update
+CREATE OR REPLACE FUNCTION public.foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ WITH u1 AS (
+ UPDATE foo foo_1 SET f1 = (foo_1.f1 + 1)
+ RETURNING old.f1,
+ old.f2,
+ old.f4,
+ new.f1,
+ new.f2,
+ new.f4
+ ), u2 AS (
+ UPDATE foo foo_1 SET f1 = (foo_1.f1 + 1)
+ RETURNING WITH (OLD AS o) o.f1,
+ o.f2,
+ o.f4,
+ new.f1,
+ new.f2,
+ new.f4
+ ), u3 AS (
+ UPDATE foo foo_1 SET f1 = (foo_1.f1 + 1)
+ RETURNING WITH (NEW AS n) old.f1,
+ old.f2,
+ old.f4,
+ n.f1,
+ n.f2,
+ n.f4
+ )
+ UPDATE foo SET f1 = (foo.f1 + 1)
+ RETURNING WITH (OLD AS o, NEW AS n) o.f1,
+ o.f2,
+ o.f4,
+ n.f1,
+ n.f2,
+ n.f4,
+ o.*::foo AS o,
+ n.*::foo AS n,
+ (o.f1 = n.f1),
+ (o.* = n.*),
+ ( SELECT (o.f2 = n.f2)),
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f1 = o.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f4 = n.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = o.*)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = n.*)) AS count;
+END
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 3014d047fe..76ed380247 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3647,7 +3647,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s
-- test deparsing
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3686,11 +3689,12 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
CREATE OR REPLACE FUNCTION public.merge_sf_test()
- RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[])
+ RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[], old_id integer, old_data text, old_filling integer[], new_id integer, new_data text, new_filling integer[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3728,12 +3732,18 @@ BEGIN ATOMIC
WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
- RETURNING MERGE_ACTION() AS action,
+ RETURNING WITH (OLD AS o, NEW AS n) MERGE_ACTION() AS action,
s.a,
s.b,
t.id,
t.data,
- t.filling;
+ t.filling,
+ o.id,
+ o.data,
+ o.filling,
+ n.id,
+ n.data,
+ n.filling;
END
CREATE FUNCTION merge_sf_test2()
RETURNS void
diff --git a/src/test/regress/expected/updatable_views.out b/src/test/regress/expected/updatable_views.out
index 8786058ed0..095df0a670 100644
--- a/src/test/regress/expected/updatable_views.out
+++ b/src/test/regress/expected/updatable_views.out
@@ -437,7 +437,8 @@ NOTICE: drop cascades to view ro_view19
-- simple updatable view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const' AS c, (SELECT concat('b: ', b)) AS d FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view1';
@@ -462,7 +463,9 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | YES
rw_view1 | b | YES
-(2 rows)
+ rw_view1 | c | NO
+ rw_view1 | d | NO
+(4 rows)
INSERT INTO rw_view1 VALUES (3, 'Row 3');
INSERT INTO rw_view1 (a) VALUES (4);
@@ -479,20 +482,22 @@ SELECT * FROM base_tbl;
5 | Unspecified
(6 rows)
+SET jit_above_cost = 0;
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+-------+---+-------------
- UPDATE | 1 | ROW 1 | 1 | ROW 1
- DELETE | 3 | ROW 3 | 3 | Row 3
- INSERT | 2 | ROW 2 | 2 | Unspecified
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
+ merge_action | a | b | old | new | a | b | c | d | a | b | c | d | a | b | c | d
+--------------+---+-------+------------------------------+----------------------------------------+---+-------+-------+----------+---+-------------+-------+----------------+---+-------------+-------+----------------
+ UPDATE | 1 | ROW 1 | (1,"Row 1",Const,"b: Row 1") | (1,"ROW 1",Const,"b: ROW 1") | 1 | Row 1 | Const | b: Row 1 | 1 | ROW 1 | Const | b: ROW 1 | 1 | ROW 1 | Const | b: ROW 1
+ DELETE | 3 | ROW 3 | (3,"Row 3",Const,"b: Row 3") | | 3 | Row 3 | Const | b: Row 3 | | | | | 3 | Row 3 | Const | b: Row 3
+ INSERT | 2 | ROW 2 | | (2,Unspecified,Const,"b: Unspecified") | | | | | 2 | Unspecified | Const | b: Unspecified | 2 | Unspecified | Const | b: Unspecified
(3 rows)
+SET jit_above_cost TO DEFAULT;
SELECT * FROM base_tbl ORDER BY a;
a | b
----+-------------
@@ -511,13 +516,13 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------------
- UPDATE | 1 | R1 | 1 | R1
- DELETE | | | 5 | Unspecified
- DELETE | 2 | R2 | 2 | Unspecified
- INSERT | 3 | R3 | 3 | Unspecified
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
+ merge_action | a | b | old | new | a | b | c | d | a | b | c | d | a | b | c | d
+--------------+---+----+----------------------------------------+----------------------------------------+---+-------------+-------+----------------+---+-------------+-------+----------------+---+-------------+-------+----------------
+ UPDATE | 1 | R1 | (1,"ROW 1",Const,"b: ROW 1") | (1,R1,Const,"b: R1") | 1 | ROW 1 | Const | b: ROW 1 | 1 | R1 | Const | b: R1 | 1 | R1 | Const | b: R1
+ DELETE | | | (5,Unspecified,Const,"b: Unspecified") | | 5 | Unspecified | Const | b: Unspecified | | | | | 5 | Unspecified | Const | b: Unspecified
+ DELETE | 2 | R2 | (2,Unspecified,Const,"b: Unspecified") | | 2 | Unspecified | Const | b: Unspecified | | | | | 2 | Unspecified | Const | b: Unspecified
+ INSERT | 3 | R3 | | (3,Unspecified,Const,"b: Unspecified") | | | | | 3 | Unspecified | Const | b: Unspecified | 3 | Unspecified | Const | b: Unspecified
(4 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -634,8 +639,10 @@ DROP TABLE base_tbl_hist;
-- view on top of view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view2';
@@ -660,27 +667,29 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view2 | aaa | YES
rw_view2 | bbb | YES
-(2 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(4 rows)
INSERT INTO rw_view2 VALUES (3, 'Row 3');
INSERT INTO rw_view2 (aaa) VALUES (4);
SELECT * FROM rw_view2;
- aaa | bbb
------+-------------
- 1 | Row 1
- 2 | Row 2
- 3 | Row 3
- 4 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Unspecified | Const1 | Const2
(4 rows)
UPDATE rw_view2 SET bbb='Row 4' WHERE aaa=4;
DELETE FROM rw_view2 WHERE aaa=2;
SELECT * FROM rw_view2;
- aaa | bbb
------+-------
- 1 | Row 1
- 3 | Row 3
- 4 | Row 4
+ aaa | bbb | c1 | c2
+-----+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Row 4 | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -688,20 +697,20 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-------------
- DELETE | 3 | R3 | 3 | Row 3
- UPDATE | 4 | R4 | 4 | R4
- INSERT | 5 | R5 | 5 | Unspecified
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+---------------------------+-------------------------------+-----+-------------+--------+--------
+ DELETE | 3 | R3 | (3,"Row 3",Const1,Const2) | | 3 | Row 3 | Const1 | Const2
+ UPDATE | 4 | R4 | (4,"Row 4",Const1,Const2) | (4,R4,Const1,Const2) | 4 | R4 | Const1 | Const2
+ INSERT | 5 | R5 | | (5,Unspecified,Const1,Const2) | 5 | Unspecified | Const1 | Const2
(3 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-------------
- 1 | Row 1
- 4 | R4
- 5 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 4 | R4 | Const1 | Const2
+ 5 | Unspecified | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -710,21 +719,21 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-----------------------
- UPDATE | | | 1 | Not matched by source
- DELETE | 4 | r4 | 4 | R4
- UPDATE | 5 | r5 | 5 | r5
- INSERT | 6 | r6 | 6 | Unspecified
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+-------------------------------+-------------------------------------------+-----+-----------------------+--------+--------
+ UPDATE | | | (1,"Row 1",Const1,Const2) | (1,"Not matched by source",Const1,Const2) | 1 | Not matched by source | Const1 | Const2
+ DELETE | 4 | r4 | (4,R4,Const1,Const2) | | 4 | R4 | Const1 | Const2
+ UPDATE | 5 | r5 | (5,Unspecified,Const1,Const2) | (5,r5,Const1,Const2) | 5 | r5 | Const1 | Const2
+ INSERT | 6 | r6 | | (6,Unspecified,Const1,Const2) | 6 | Unspecified | Const1 | Const2
(4 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-----------------------
- 1 | Not matched by source
- 5 | r5
- 6 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-----------------------+--------+--------
+ 1 | Not matched by source | Const1 | Const2
+ 5 | r5 | Const1 | Const2
+ 6 | Unspecified | Const1 | Const2
(3 rows)
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -886,16 +895,25 @@ SELECT table_name, column_name, is_updatable
rw_view2 | b | YES
(4 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | a | b
+---+---+---+-------
+ | | 3 | Row 3
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+ a | b | a | b
+---+----+---+----
+ 3 | R3 | 3 | R3
+(1 row)
+
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+----+---+-----------
+ 3 | R3 | 3 | Row three
(1 row)
SELECT * FROM rw_view2;
@@ -906,10 +924,10 @@ SELECT * FROM rw_view2;
3 | Row three
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+-----------+---+---
+ 3 | Row three | |
(1 row)
SELECT * FROM rw_view2;
@@ -960,8 +978,10 @@ drop cascades to view rw_view2
-- view on top of view with triggers
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name LIKE 'rw_view%'
@@ -992,9 +1012,12 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE FUNCTION rw_view1_trig_fn()
RETURNS trigger AS
@@ -1002,9 +1025,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -1045,9 +1070,12 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_upd_trig INSTEAD OF UPDATE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1081,9 +1109,12 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_del_trig INSTEAD OF DELETE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1117,41 +1148,44 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+---+----+----+---+-------+----------------+--------
+ | | | | 3 | Row 3 | Trigger Const1 | Const2
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-------+--------+--------+---+-----------+----------------+--------
+ 3 | Row 3 | Const1 | Const2 | 3 | Row three | Trigger Const1 | Const2
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-----------
- 1 | Row 1
- 2 | Row 2
- 3 | Row three
+ a | b | c1 | c2
+---+-----------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row three | Const1 | Const2
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-----------+--------+--------+---+---+----+----
+ 3 | Row three | Const1 | Const2 | | | |
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-------
- 1 | Row 1
- 2 | Row 2
+ a | b | c1 | c2
+---+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
(2 rows)
MERGE INTO rw_view2 t
@@ -1159,12 +1193,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------
- DELETE | 1 | R1 | 1 | Row 1
- UPDATE | 2 | R2 | 2 | R2
- INSERT | 3 | R3 | 3 | R3
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+---------------------------+--------------------------------+---+-------+----------------+--------
+ DELETE | 1 | R1 | (1,"Row 1",Const1,Const2) | | 1 | Row 1 | Const1 | Const2
+ UPDATE | 2 | R2 | (2,"Row 2",Const1,Const2) | (2,R2,"Trigger Const1",Const2) | 2 | R2 | Trigger Const1 | Const2
+ INSERT | 3 | R3 | | (3,R3,"Trigger Const1",Const2) | 3 | R3 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -1182,12 +1216,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-----------------------
- UPDATE | 2 | r2 | 2 | r2
- UPDATE | | | 3 | Not matched by source
- INSERT | 1 | r1 | 1 | r1
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+----------------------+-----------------------------------------------------+---+-----------------------+----------------+--------
+ UPDATE | 2 | r2 | (2,R2,Const1,Const2) | (2,r2,"Trigger Const1",Const2) | 2 | r2 | Trigger Const1 | Const2
+ UPDATE | | | (3,R3,Const1,Const2) | (3,"Not matched by source","Trigger Const1",Const2) | 3 | Not matched by source | Trigger Const1 | Const2
+ INSERT | 1 | r1 | | (1,r1,"Trigger Const1",Const2) | 1 | r1 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql
index 54929a92fa..07b6295b3b 100644
--- a/src/test/regress/sql/merge.sql
+++ b/src/test/regress/sql/merge.sql
@@ -235,7 +235,7 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -677,7 +677,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 THEN
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -930,7 +930,9 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -956,7 +958,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -970,7 +972,7 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -988,7 +990,7 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
ROLLBACK;
@@ -1265,7 +1267,7 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
@@ -1456,7 +1458,7 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
diff --git a/src/test/regress/sql/returning.sql b/src/test/regress/sql/returning.sql
index a460f82fb7..4b761a4397 100644
--- a/src/test/regress/sql/returning.sql
+++ b/src/test/regress/sql/returning.sql
@@ -160,3 +160,212 @@ INSERT INTO foo AS bar DEFAULT VALUES RETURNING *; -- ok
INSERT INTO foo AS bar DEFAULT VALUES RETURNING foo.*; -- fails, wrong name
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.*; -- ok
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.f3; -- ok
+
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DROP TABLE zerocol;
+
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DROP TABLE foo_parted;
+
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ WITH u1 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING old.*, new.*
+ ), u2 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (OLD AS o) o.*, new.*
+ ), u3 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (NEW AS n) old.*, n.*
+ )
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.*, n.*, o, n, o.f1 = n.f1, o = n,
+ (SELECT o.f2 = n.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = n.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = n);
+END;
+
+\sf foo_update
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 4a5fa50585..fdd3ff1d16 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1294,7 +1294,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -1333,7 +1336,8 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
diff --git a/src/test/regress/sql/updatable_views.sql b/src/test/regress/sql/updatable_views.sql
index 93b693ae83..c071fffc11 100644
--- a/src/test/regress/sql/updatable_views.sql
+++ b/src/test/regress/sql/updatable_views.sql
@@ -154,7 +154,8 @@ DROP SEQUENCE uv_seq CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const' AS c, (SELECT concat('b: ', b)) AS d FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -175,13 +176,18 @@ UPDATE rw_view1 SET a=5 WHERE a=4;
DELETE FROM rw_view1 WHERE b='Row 2';
SELECT * FROM base_tbl;
+SET jit_above_cost = 0;
+
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
+
+SET jit_above_cost TO DEFAULT;
+
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view1 t
@@ -191,7 +197,7 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view1 SET a=6 WHERE a=5;
@@ -240,8 +246,10 @@ DROP TABLE base_tbl_hist;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -268,7 +276,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
MERGE INTO rw_view2 t
@@ -277,7 +285,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -362,10 +370,14 @@ SELECT table_name, column_name, is_updatable
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t USING (VALUES (3, 'Row 3')) AS v(a,b) ON t.a = v.a
@@ -381,8 +393,10 @@ DROP TABLE base_tbl CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -407,9 +421,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -479,10 +495,10 @@ SELECT table_name, column_name, is_updatable
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t
@@ -490,7 +506,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view2 t
@@ -498,7 +514,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e1c4f913f8..61ec01dcdc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2474,6 +2474,10 @@ RestrictInfo
Result
ResultRelInfo
ResultState
+ReturningClause
+ReturningExpr
+ReturningOption
+ReturningOptionKind
ReturnSetInfo
ReturnStmt
RevmapContents
@@ -2624,6 +2628,7 @@ SetOperationStmt
SetQuantifier
SetToDefault
SetupWorkerPtrType
+SetVarReturningType_context
ShDependObjectInfo
SharedAggInfo
SharedBitmapHeapInstrumentation
@@ -3095,6 +3100,7 @@ VariableSpace
VariableStatData
VariableSubstituteHook
Variables
+VarReturningType
Vector32
Vector8
VersionedQuery
--
2.43.0
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2025-01-03 19:39 ` Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Robert Treat @ 2025-01-03 19:39 UTC (permalink / raw)
To: Dean Rasheed <[email protected]>; +Cc: jian he <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Wed, Jan 1, 2025 at 3:20 AM Dean Rasheed <[email protected]> wrote:
> On Thu, 28 Nov 2024 at 11:45, Dean Rasheed <[email protected]> wrote:
> Attached is an updated patch with some additional tidying up, plus the
> following changes:
>
Hey Dean,
This is really nice work. I was curious what you think the status of
this patch is at this point and if you are still thinking of
committing it for v18?
Robert Treat
https://xzilla.net
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
@ 2025-01-07 10:56 ` Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2025-01-07 10:56 UTC (permalink / raw)
To: Robert Treat <[email protected]>; +Cc: jian he <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Fri, 3 Jan 2025 at 19:39, Robert Treat <[email protected]> wrote:
>
> This is really nice work. I was curious what you think the status of
> this patch is at this point and if you are still thinking of
> committing it for v18?
>
Thanks for looking. I think that the patch is in good shape, and it
has had a decent amount of testing and review, so yes, I do plan to
commit it for v18.
In fact I think it is probably best to commit it soon, so that it has
more time in tree, ahead of v18. With that in mind, I plan to go over
it again in detail, and barring any other review comments, commit it.
Regards,
Dean
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2025-01-08 06:16 ` jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: jian he @ 2025-01-08 06:16 UTC (permalink / raw)
To: Dean Rasheed <[email protected]>; +Cc: Robert Treat <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
hi.
two minor issues.
if (qry->returningList == NIL)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("RETURNING must have at least one column"),
parser_errposition(pstate,
exprLocation(linitial(returningClause->exprs)))));
we can reduce one level parenthesis by:
if (qry->returningList == NIL)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("RETURNING must have at least one column"),
parser_errposition(pstate,
exprLocation(linitial(returningClause->exprs))));
seems no tests for this error case.
we can add one in case someone in future is wondering if this is ever reachable.
like:
create temp table s1();
insert into s1 default values returning new.*;
drop temp table s1;
transformReturningClause
case RETURNING_OPTION_NEW: not tested,
we can add one at src/test/regress/sql/returning.sql line 176:
INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS n, old AS o, new as n) *;
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
@ 2025-01-08 09:38 ` Dean Rasheed <[email protected]>
2025-01-16 15:28 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2025-01-08 09:38 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Robert Treat <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Wed, 8 Jan 2025 at 06:17, jian he <[email protected]> wrote:
>
> hi.
> two minor issues.
>
> if (qry->returningList == NIL)
> ereport(ERROR,
> (errcode(ERRCODE_SYNTAX_ERROR),
> errmsg("RETURNING must have at least one column"),
> parser_errposition(pstate,
>
> exprLocation(linitial(returningClause->exprs)))));
>
> we can reduce one level parenthesis
I don't think that's a good idea in this case. That's a pre-existing
error check in which all this patch does is replace "returningList"
with "returningClause->exprs". Removing the unneeded level of
parentheses would change a 1-line diff into a 5-line diff, making it
harder to see what the patch had actually changed.
In general, the code-base is littered with ereport()'s that have an
extra, now-unnecessary, level of parentheses. I don't think it should
be the job of every patch to try to fix those up, though I do think we
should try to avoid them when adding new ereport()'s.
> seems no tests for this error case.
> we can add one in case someone in future is wondering if this is ever reachable.
> like:
> create temp table s1();
> insert into s1 default values returning new.*;
> drop temp table s1;
Hmm, I was tempted to say that that's also just a pre-existing case,
but on reflection, I think it's probably worth confirming that "old.*,
new.*, *" can expand to zero columns. I had already added a set of
zero-column table tests in returning.sql, so I just added another test
case to that.
> transformReturningClause
> case RETURNING_OPTION_NEW: not tested,
> we can add one at src/test/regress/sql/returning.sql line 176:
>
> INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS n, old AS o, new as n) *;
OK, done.
I also noticed that I had failed to use quote_identifier() in
ruleutils.c for the new WITH aliases, so I've fixed that and adjusted
a couple of the test cases to test that.
It looks to me as though there might be pre-existing bugs in that
area, so I'll go investigate and start a new thread, if that's the
case.
Regards,
Dean
Attachments:
[text/x-patch] v23-0001-Add-OLD-NEW-support-to-RETURNING-in-DML-queries.patch (243.8K, ../../CAEZATCV61iAaL=jeAP=KpbgL+OQzeKo8KOeKtr0mG=2x6nz6fg@mail.gmail.com/2-v23-0001-Add-OLD-NEW-support-to-RETURNING-in-DML-queries.patch)
download | inline diff:
From 2fb2d89008b6809cc3a8b8bea62c2faad72064dd Mon Sep 17 00:00:00 2001
From: Dean Rasheed <[email protected]>
Date: Mon, 23 Dec 2024 14:15:15 +0000
Subject: [PATCH v23] Add OLD/NEW support to RETURNING in DML queries.
This allows the RETURNING list of INSERT/UPDATE/DELETE/MERGE queries
to explicitly return old and new values by using the special aliases
"old" and "new", which are automatically added to the query (if not
already defined) while parsing its RETURNING list, allowing things
like:
RETURNING old.colname, new.colname, ...
RETURNING old.*, new.*
Additionally, a new syntax is supported, allowing the names "old" and
"new" to be changed to user-supplied alias names, e.g.:
RETURNING WITH (OLD AS o, NEW AS n) o.colname, n.colname, ...
This is useful when the names "old" and "new" are already defined,
such as inside trigger functions, allowing backwards compatibility to
be maintained -- the interpretation of any existing queries that
happen to already refer to relations called "old" or "new", or use
those as aliases for other relations, is not changed.
For an INSERT, old values will generally be NULL, and for a DELETE,
new values will generally be NULL, but that may change for an INSERT
with an ON CONFLICT ... DO UPDATE clause, or if a query rewrite rule
changes the command type. Therefore, we put no restrictions on the use
of old and new in any DML queries.
Dean Rasheed, reviewed by Jian He and Jeff Davis.
Discussion: https://postgr.es/m/CAEZATCWx0J0-v=Qjc6gXzR=KtsdvAE7Ow=D=mu50AgOe+pvisQ@mail.gmail.com
---
.../postgres_fdw/expected/postgres_fdw.out | 124 +++-
contrib/postgres_fdw/sql/postgres_fdw.sql | 25 +-
doc/src/sgml/dml.sgml | 41 +-
doc/src/sgml/ref/delete.sgml | 40 +-
doc/src/sgml/ref/insert.sgml | 54 +-
doc/src/sgml/ref/merge.sgml | 35 +-
doc/src/sgml/ref/update.sgml | 38 +-
doc/src/sgml/rules.sgml | 17 +
src/backend/executor/execExpr.c | 149 ++++-
src/backend/executor/execExprInterp.c | 199 ++++++-
src/backend/executor/execMain.c | 1 +
src/backend/executor/execUtils.c | 28 +
src/backend/executor/nodeModifyTable.c | 223 ++++++-
src/backend/jit/llvm/llvmjit_expr.c | 119 +++-
src/backend/nodes/makefuncs.c | 12 +-
src/backend/nodes/nodeFuncs.c | 46 +-
src/backend/optimizer/path/allpaths.c | 1 +
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/plan/setrefs.c | 15 +
src/backend/optimizer/plan/subselect.c | 45 +-
src/backend/optimizer/prep/prepjointree.c | 3 +-
src/backend/optimizer/util/appendinfo.c | 21 +-
src/backend/optimizer/util/clauses.c | 3 +
src/backend/optimizer/util/paramassign.c | 47 ++
src/backend/optimizer/util/plancat.c | 4 +-
src/backend/optimizer/util/var.c | 44 ++
src/backend/parser/analyze.c | 150 ++++-
src/backend/parser/gram.y | 57 +-
src/backend/parser/parse_clause.c | 2 +
src/backend/parser/parse_expr.c | 18 +-
src/backend/parser/parse_merge.c | 4 +-
src/backend/parser/parse_relation.c | 33 +-
src/backend/parser/parse_target.c | 4 +-
src/backend/rewrite/rewriteHandler.c | 9 +
src/backend/rewrite/rewriteManip.c | 128 +++-
src/backend/utils/adt/ruleutils.c | 113 +++-
src/include/executor/execExpr.h | 25 +-
src/include/executor/executor.h | 1 +
src/include/nodes/execnodes.h | 16 +
src/include/nodes/parsenodes.h | 52 +-
src/include/nodes/plannodes.h | 2 +
src/include/nodes/primnodes.h | 40 ++
src/include/optimizer/optimizer.h | 1 +
src/include/optimizer/paramassign.h | 2 +
src/include/parser/analyze.h | 5 +-
src/include/parser/parse_node.h | 7 +
src/include/parser/parse_relation.h | 1 +
src/include/rewrite/rewriteManip.h | 1 +
src/interfaces/ecpg/preproc/parse.pl | 4 +-
src/test/isolation/expected/merge-update.out | 52 +-
src/test/isolation/specs/merge-update.spec | 4 +-
src/test/regress/expected/merge.out | 96 +--
src/test/regress/expected/returning.out | 548 ++++++++++++++++++
src/test/regress/expected/rules.out | 20 +-
src/test/regress/expected/updatable_views.out | 240 ++++----
src/test/regress/sql/merge.sql | 18 +-
src/test/regress/sql/returning.sql | 211 +++++++
src/test/regress/sql/rules.sql | 8 +-
src/test/regress/sql/updatable_views.sql | 50 +-
src/tools/pgindent/typedefs.list | 6 +
60 files changed, 2894 insertions(+), 388 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index bf322198a2..687367896b 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -4975,12 +4975,12 @@ INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
- c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
-------+-----+-----+----+----+----+------------+----
- 1101 | 201 | aaa | | | | ft2 |
- 1102 | 202 | bbb | | | | ft2 |
- 1103 | 203 | ccc | | | | ft2 |
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
+ old | new | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+-----+---------------------------------+----+----+----+----+----+----+----+----+------+-----+-----+----+----+----+------------+----
+ | (1101,201,aaa,,,,"ft2 ",) | | | | | | | | | 1101 | 201 | aaa | | | | ft2 |
+ | (1102,202,bbb,,,,"ft2 ",) | | | | | | | | | 1102 | 202 | bbb | | | | ft2 |
+ | (1103,203,ccc,,,,"ft2 ",) | | | | | | | | | 1103 | 203 | ccc | | | | ft2 |
(3 rows)
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
@@ -5111,6 +5111,31 @@ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING
1017 | 507 | 0001700017_update7 | | | | ft2 |
(102 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.c1, old.c2, old.c3, old.c4, old.c5, old.c6, old.c7, old.c8, new.c1, new.c2, new.c3, new.c4, new.c5, new.c6, new.c7, new.c8
+ Remote SQL: UPDATE "S 1"."T 1" SET c2 = $2, c3 = $3 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan on public.ft2
+ Output: (c2 + 400), (c3 || '_update7b'::text), ctid, ft2.*
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 7)) FOR UPDATE
+(6 rows)
+
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8
+----+-----+---------------+------------------------------+--------------------------+----+------------+-----+----+-----+------------------------+------------------------------+--------------------------+----+------------+-----
+ 7 | 407 | 00007_update7 | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo | 7 | 807 | 00007_update7_update7b | Thu Jan 08 00:00:00 1970 PST | Thu Jan 08 00:00:00 1970 | 7 | 7 | foo
+ 17 | 407 | 00017_update7 | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo | 17 | 807 | 00017_update7_update7b | Sun Jan 18 00:00:00 1970 PST | Sun Jan 18 00:00:00 1970 | 7 | 7 | foo
+ 27 | 407 | 00027_update7 | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo | 27 | 807 | 00027_update7_update7b | Wed Jan 28 00:00:00 1970 PST | Wed Jan 28 00:00:00 1970 | 7 | 7 | foo
+ 37 | 407 | 00037_update7 | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo | 37 | 807 | 00037_update7_update7b | Sat Feb 07 00:00:00 1970 PST | Sat Feb 07 00:00:00 1970 | 7 | 7 | foo
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -5241,6 +5266,29 @@ DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4;
1105 |
(103 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------
+ Delete on public.ft2
+ Output: old.c1, c4
+ Remote SQL: DELETE FROM "S 1"."T 1" WHERE ctid = $1 RETURNING "C 1", c4
+ -> Foreign Scan on public.ft2
+ Output: ctid
+ Remote SQL: SELECT ctid FROM "S 1"."T 1" WHERE (("C 1" < 40)) AND ((("C 1" % 10) = 6)) FOR UPDATE
+(6 rows)
+
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ c1 | c4
+----+------------------------------
+ 6 | Wed Jan 07 00:00:00 1970 PST
+ 16 | Sat Jan 17 00:00:00 1970 PST
+ 26 | Tue Jan 27 00:00:00 1970 PST
+ 36 | Fri Feb 06 00:00:00 1970 PST
+(4 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
QUERY PLAN
@@ -6165,6 +6213,70 @@ UPDATE ft2 SET c3 = 'foo'
(1296,96,foo,,,,"ft2 ",) | 1296 | 96 | foo | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
(16 rows)
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on public.ft2
+ Output: old.*, new.*, ft2.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: UPDATE "S 1"."T 1" SET c3 = $2 WHERE ctid = $1 RETURNING "C 1", c2, c3, c4, c5, c6, c7, c8
+ -> Foreign Scan
+ Output: 'bar'::text, ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Relations: ((public.ft2) INNER JOIN (public.ft4)) INNER JOIN (public.ft5)
+ Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2.c1, r2.c2, r2.c3) END, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r2.c1, r2.c2, r2.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r2 ON (((r1.c2 = r2.c1)) AND ((r1."C 1" > 1200)))) INNER JOIN "S 1"."T 4" r3 ON (((r2.c1 = r3.c1)))) FOR UPDATE OF r1
+ -> Nested Loop
+ Output: ft2.ctid, ft2.*, ft4.*, ft5.*, ft4.c1, ft4.c2, ft4.c3
+ Join Filter: (ft4.c1 = ft5.c1)
+ -> Sort
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Sort Key: ft2.c2
+ -> Hash Join
+ Output: ft2.ctid, ft2.*, ft2.c2, ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Hash Cond: (ft2.c2 = ft4.c1)
+ -> Foreign Scan on public.ft2
+ Output: ft2.ctid, ft2.*, ft2.c2
+ Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8, ctid FROM "S 1"."T 1" WHERE (("C 1" > 1200)) FOR UPDATE
+ -> Hash
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ -> Foreign Scan on public.ft4
+ Output: ft4.*, ft4.c1, ft4.c2, ft4.c3
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 3"
+ -> Materialize
+ Output: ft5.*, ft5.c1
+ -> Foreign Scan on public.ft5
+ Output: ft5.*, ft5.c1
+ Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 4"
+(29 rows)
+
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ old | new | ft2 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | ft4 | c1 | c2 | c3
+--------------------------------+--------------------------------+--------------------------------+------+----+-----+----+----+----+------------+----+----------------+----+----+--------
+ (1206,6,foo,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | (1206,6,bar,,,,"ft2 ",) | 1206 | 6 | bar | | | | ft2 | | (6,7,AAA006) | 6 | 7 | AAA006
+ (1212,12,foo,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | (1212,12,bar,,,,"ft2 ",) | 1212 | 12 | bar | | | | ft2 | | (12,13,AAA012) | 12 | 13 | AAA012
+ (1224,24,foo,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | (1224,24,bar,,,,"ft2 ",) | 1224 | 24 | bar | | | | ft2 | | (24,25,AAA024) | 24 | 25 | AAA024
+ (1230,30,foo,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | (1230,30,bar,,,,"ft2 ",) | 1230 | 30 | bar | | | | ft2 | | (30,31,AAA030) | 30 | 31 | AAA030
+ (1242,42,foo,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | (1242,42,bar,,,,"ft2 ",) | 1242 | 42 | bar | | | | ft2 | | (42,43,AAA042) | 42 | 43 | AAA042
+ (1248,48,foo,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | (1248,48,bar,,,,"ft2 ",) | 1248 | 48 | bar | | | | ft2 | | (48,49,AAA048) | 48 | 49 | AAA048
+ (1260,60,foo,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | (1260,60,bar,,,,"ft2 ",) | 1260 | 60 | bar | | | | ft2 | | (60,61,AAA060) | 60 | 61 | AAA060
+ (1266,66,foo,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | (1266,66,bar,,,,"ft2 ",) | 1266 | 66 | bar | | | | ft2 | | (66,67,AAA066) | 66 | 67 | AAA066
+ (1278,78,foo,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | (1278,78,bar,,,,"ft2 ",) | 1278 | 78 | bar | | | | ft2 | | (78,79,AAA078) | 78 | 79 | AAA078
+ (1284,84,foo,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | (1284,84,bar,,,,"ft2 ",) | 1284 | 84 | bar | | | | ft2 | | (84,85,AAA084) | 84 | 85 | AAA084
+ (1296,96,foo,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | (1296,96,bar,,,,"ft2 ",) | 1296 | 96 | bar | | | | ft2 | | (96,97,AAA096) | 96 | 97 | AAA096
+ (1218,18,foo,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | (1218,18,bar,,,,"ft2 ",) | 1218 | 18 | bar | | | | ft2 | | (18,19,AAA018) | 18 | 19 | AAA018
+ (1236,36,foo,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | (1236,36,bar,,,,"ft2 ",) | 1236 | 36 | bar | | | | ft2 | | (36,37,AAA036) | 36 | 37 | AAA036
+ (1254,54,foo,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | (1254,54,bar,,,,"ft2 ",) | 1254 | 54 | bar | | | | ft2 | | (54,55,AAA054) | 54 | 55 | AAA054
+ (1272,72,foo,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | (1272,72,bar,,,,"ft2 ",) | 1272 | 72 | bar | | | | ft2 | | (72,73,AAA072) | 72 | 73 | AAA072
+ (1290,90,foo,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | (1290,90,bar,,,,"ft2 ",) | 1290 | 90 | bar | | | | ft2 | | (90,91,AAA090) | 90 | 91 | AAA090
+(16 rows)
+
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 3900522ccb..b58ab6ee58 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -1469,7 +1469,7 @@ EXPLAIN (verbose, costs off)
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3) SELECT c1+1000,c2+100, c3 || c3 FROM ft2 LIMIT 20;
INSERT INTO ft2 (c1,c2,c3)
- VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING *;
+ VALUES (1101,201,'aaa'), (1102,202,'bbb'), (1103,203,'ccc') RETURNING old, new, old.*, new.*;
INSERT INTO ft2 (c1,c2,c3) VALUES (1104,204,'ddd'), (1105,205,'eee');
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 300, c3 = c3 || '_update3' WHERE c1 % 10 = 3; -- can be pushed down
@@ -1477,6 +1477,13 @@ UPDATE ft2 SET c2 = c2 + 300, c3 = c3 || '_update3' WHERE c1 % 10 = 3;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *; -- can be pushed down
UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7' WHERE c1 % 10 = 7 RETURNING *;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*; -- can't be pushed down
+ UPDATE ft2 SET c2 = c2 + 400, c3 = c3 || '_update7b' WHERE c1 % 10 = 7 AND c1 < 40
+ RETURNING old.*, new.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
FROM ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 9; -- can be pushed down
@@ -1485,6 +1492,11 @@ UPDATE ft2 SET c2 = ft2.c2 + 500, c3 = ft2.c3 || '_update9', c7 = DEFAULT
EXPLAIN (verbose, costs off)
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4; -- can be pushed down
DELETE FROM ft2 WHERE c1 % 10 = 5 RETURNING c1, c4;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4; -- can't be pushed down
+ DELETE FROM ft2 WHERE c1 % 10 = 6 AND c1 < 40 RETURNING old.c1, c4;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2; -- can be pushed down
DELETE FROM ft2 USING ft1 WHERE ft1.c1 = ft2.c2 AND ft1.c1 % 10 = 2;
@@ -1511,6 +1523,17 @@ UPDATE ft2 SET c3 = 'foo'
FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
RETURNING ft2, ft2.*, ft4, ft4.*;
+BEGIN;
+ EXPLAIN (verbose, costs off)
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*; -- can't be pushed down
+ UPDATE ft2 SET c3 = 'bar'
+ FROM ft4 INNER JOIN ft5 ON (ft4.c1 = ft5.c1)
+ WHERE ft2.c1 > 1200 AND ft2.c2 = ft4.c1
+ RETURNING old, new, ft2, ft2.*, ft4, ft4.*;
+ROLLBACK;
EXPLAIN (verbose, costs off)
DELETE FROM ft2
USING ft4 LEFT JOIN ft5 ON (ft4.c1 = ft5.c1)
diff --git a/doc/src/sgml/dml.sgml b/doc/src/sgml/dml.sgml
index 3d95bdb94e..458aee788b 100644
--- a/doc/src/sgml/dml.sgml
+++ b/doc/src/sgml/dml.sgml
@@ -308,7 +308,8 @@ DELETE FROM products;
</para>
<para>
- In an <command>INSERT</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>INSERT</command>, the default data available to
+ <literal>RETURNING</literal> is
the row as it was inserted. This is not so useful in trivial inserts,
since it would just repeat the data provided by the client. But it can
be very handy when relying on computed default values. For example,
@@ -325,7 +326,8 @@ INSERT INTO users (firstname, lastname) VALUES ('Joe', 'Cool') RETURNING id;
</para>
<para>
- In an <command>UPDATE</command>, the data available to <literal>RETURNING</literal> is
+ In an <command>UPDATE</command>, the default data available to
+ <literal>RETURNING</literal> is
the new content of the modified row. For example:
<programlisting>
UPDATE products SET price = price * 1.10
@@ -335,7 +337,8 @@ UPDATE products SET price = price * 1.10
</para>
<para>
- In a <command>DELETE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>DELETE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the deleted row. For example:
<programlisting>
DELETE FROM products
@@ -345,7 +348,8 @@ DELETE FROM products
</para>
<para>
- In a <command>MERGE</command>, the data available to <literal>RETURNING</literal> is
+ In a <command>MERGE</command>, the default data available to
+ <literal>RETURNING</literal> is
the content of the source row plus the content of the inserted, updated, or
deleted target row. Since it is quite common for the source and target to
have many of the same columns, specifying <literal>RETURNING *</literal>
@@ -359,6 +363,35 @@ MERGE INTO products p USING new_products n ON p.product_no = n.product_no
</programlisting>
</para>
+ <para>
+ In each of these commands, it is also possible to explicitly return the
+ old and new content of the modified row. For example:
+<programlisting>
+UPDATE products SET price = price * 1.10
+ WHERE price <= 99.99
+ RETURNING name, old.price AS old_price, new.price AS new_price,
+ new.price - old.price AS price_change;
+</programlisting>
+ In this example, writing <literal>new.price</literal> is the same as
+ just writing <literal>price</literal>, but it makes the meaning clearer.
+ </para>
+
+ <para>
+ This syntax for returning old and new values is available in
+ <command>INSERT</command>, <command>UPDATE</command>,
+ <command>DELETE</command>, and <command>MERGE</command> commands, but
+ typically old values will be <literal>NULL</literal> for an
+ <command>INSERT</command>, and new values will be <literal>NULL</literal>
+ for a <command>DELETE</command>. However, there are situations where it
+ can still be useful for those commands. For example, in an
+ <command>INSERT</command> with an
+ <link linkend="sql-on-conflict"><literal>ON CONFLICT DO UPDATE</literal></link>
+ clause, the old values will be non-<literal>NULL</literal> for conflicting
+ rows. Similarly, if a <command>DELETE</command> is turned into an
+ <command>UPDATE</command> by a <link linkend="sql-createrule">rewrite rule</link>,
+ the new values may be non-<literal>NULL</literal>.
+ </para>
+
<para>
If there are triggers (<xref linkend="triggers"/>) on the target table,
the data available to <literal>RETURNING</literal> is the row as modified by
diff --git a/doc/src/sgml/ref/delete.sgml b/doc/src/sgml/ref/delete.sgml
index 7717855bc9..29649f6afd 100644
--- a/doc/src/sgml/ref/delete.sgml
+++ b/doc/src/sgml/ref/delete.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
[ USING <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -160,6 +161,26 @@ DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ *
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -170,6 +191,23 @@ DELETE FROM [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ *
or table(s) listed in <literal>USING</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return old values.
+ </para>
+
+ <para>
+ For a simple <command>DELETE</command>, all new values will be
+ <literal>NULL</literal>. However, if an <literal>ON DELETE</literal>
+ rule causes an <command>INSERT</command> or <command>UPDATE</command>
+ to be executed instead, the new values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml
index 6f0adee1a1..3f13991779 100644
--- a/doc/src/sgml/ref/insert.sgml
+++ b/doc/src/sgml/ref/insert.sgml
@@ -26,7 +26,8 @@ INSERT INTO <replaceable class="parameter">table_name</replaceable> [ AS <replac
[ OVERRIDING { SYSTEM | USER } VALUE ]
{ DEFAULT VALUES | VALUES ( { <replaceable class="parameter">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="parameter">query</replaceable> }
[ ON CONFLICT [ <replaceable class="parameter">conflict_target</replaceable> ] <replaceable class="parameter">conflict_action</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">conflict_target</replaceable> can be one of:</phrase>
@@ -293,6 +294,26 @@ INSERT INTO <replaceable class="parameter">table_name</replaceable> [ AS <replac
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -305,6 +326,23 @@ INSERT INTO <replaceable class="parameter">table_name</replaceable> [ AS <replac
<literal>*</literal> to return all columns of the inserted or updated
row(s).
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values.
+ </para>
+
+ <para>
+ For a simple <command>INSERT</command>, all old values will be
+ <literal>NULL</literal>. However, for an <command>INSERT</command>
+ with an <literal>ON CONFLICT DO UPDATE</literal> clause, the old
+ values may be non-<literal>NULL</literal>.
+ </para>
</listitem>
</varlistentry>
@@ -711,6 +749,20 @@ INSERT INTO employees_log SELECT *, current_timestamp FROM upd;
INSERT INTO distributors (did, dname)
VALUES (5, 'Gizmo Transglobal'), (6, 'Associated Computing, Inc')
ON CONFLICT (did) DO UPDATE SET dname = EXCLUDED.dname;
+</programlisting>
+ </para>
+ <para>
+ Insert or update new distributors as above, returning information
+ about any existing values that were updated, together with the new data
+ inserted. Note that the returned values for <literal>old_did</literal>
+ and <literal>old_dname</literal> will be <literal>NULL</literal> for
+ non-conflicting rows:
+<programlisting>
+INSERT INTO distributors (did, dname)
+ VALUES (5, 'Gizmo Transglobal'), (6, 'Associated Computing, Inc')
+ ON CONFLICT (did) DO UPDATE SET dname = EXCLUDED.dname
+ RETURNING old.did AS old_did, old.dname AS old_dname,
+ new.did AS new_did, new.dname AS new_dname;
</programlisting>
</para>
<para>
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
index d80a5c5cc9..3da0cd9e8e 100644
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -25,7 +25,8 @@ PostgreSQL documentation
MERGE INTO [ ONLY ] <replaceable class="parameter">target_table_name</replaceable> [ * ] [ [ AS ] <replaceable class="parameter">target_alias</replaceable> ]
USING <replaceable class="parameter">data_source</replaceable> ON <replaceable class="parameter">join_condition</replaceable>
<replaceable class="parameter">when_clause</replaceable> [...]
-[ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+[ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
<phrase>where <replaceable class="parameter">data_source</replaceable> is:</phrase>
@@ -499,6 +500,25 @@ DELETE
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -517,6 +537,17 @@ DELETE
qualifying the <literal>*</literal> with the name or alias of the source
or target table.
</para>
+ <para>
+ A column name or <literal>*</literal> may also be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values from the target table to be returned. An unqualified column
+ name, or <literal>*</literal>, or a column name or <literal>*</literal>
+ qualified using the target table name or alias will return new values
+ for <literal>INSERT</literal> and <literal>UPDATE</literal> actions, and
+ old values for <literal>DELETE</literal> actions.
+ </para>
</listitem>
</varlistentry>
@@ -722,7 +753,7 @@ WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN
UPDATE SET stock = w.stock + s.stock_delta
WHEN MATCHED THEN
DELETE
-RETURNING merge_action(), w.*;
+RETURNING merge_action(), w.winename, old.stock AS old_stock, new.stock AS new_stock;
</programlisting>
The <literal>wine_stock_changes</literal> table might be, for example, a
diff --git a/doc/src/sgml/ref/update.sgml b/doc/src/sgml/ref/update.sgml
index 1c433bec2b..12ec5ba070 100644
--- a/doc/src/sgml/ref/update.sgml
+++ b/doc/src/sgml/ref/update.sgml
@@ -29,7 +29,8 @@ UPDATE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [
} [, ...]
[ FROM <replaceable class="parameter">from_item</replaceable> [, ...] ]
[ WHERE <replaceable class="parameter">condition</replaceable> | WHERE CURRENT OF <replaceable class="parameter">cursor_name</replaceable> ]
- [ RETURNING { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
+ [ RETURNING [ WITH ( { OLD | NEW } AS <replaceable class="parameter">output_alias</replaceable> [, ...] ) ]
+ { * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] } [, ...] ]
</synopsis>
</refsynopsisdiv>
@@ -211,6 +212,26 @@ UPDATE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">output_alias</replaceable></term>
+ <listitem>
+ <para>
+ An optional substitute name for <literal>OLD</literal> or
+ <literal>NEW</literal> rows in the <literal>RETURNING</literal> list.
+ </para>
+
+ <para>
+ By default, old values from the target table can be returned by writing
+ <literal>OLD.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>OLD.*</literal>, and new values can be returned by writing
+ <literal>NEW.<replaceable class="parameter">column_name</replaceable></literal>
+ or <literal>NEW.*</literal>. When an alias is provided, these names are
+ hidden and the old or new rows must be referred to using the alias.
+ For example <literal>RETURNING WITH (OLD AS o, NEW AS n) o.*, n.*</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">output_expression</replaceable></term>
<listitem>
@@ -221,6 +242,16 @@ UPDATE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [
or table(s) listed in <literal>FROM</literal>.
Write <literal>*</literal> to return all columns.
</para>
+
+ <para>
+ A column name or <literal>*</literal> may be qualified using
+ <literal>OLD</literal> or <literal>NEW</literal>, or the corresponding
+ <replaceable class="parameter">output_alias</replaceable> for
+ <literal>OLD</literal> or <literal>NEW</literal>, to cause old or new
+ values to be returned. An unqualified column name, or
+ <literal>*</literal>, or a column name or <literal>*</literal> qualified
+ using the target table name or alias will return new values.
+ </para>
</listitem>
</varlistentry>
@@ -348,12 +379,13 @@ UPDATE weather SET temp_lo = temp_lo+1, temp_hi = temp_lo+15, prcp = DEFAULT
</para>
<para>
- Perform the same operation and return the updated entries:
+ Perform the same operation and return the updated entries, and the old
+ precipitation value:
<programlisting>
UPDATE weather SET temp_lo = temp_lo+1, temp_hi = temp_lo+15, prcp = DEFAULT
WHERE city = 'San Francisco' AND date = '2003-07-03'
- RETURNING temp_lo, temp_hi, prcp;
+ RETURNING temp_lo, temp_hi, prcp, old.prcp AS old_prcp;
</programlisting>
</para>
diff --git a/doc/src/sgml/rules.sgml b/doc/src/sgml/rules.sgml
index 7a928bd7b9..e992baa91c 100644
--- a/doc/src/sgml/rules.sgml
+++ b/doc/src/sgml/rules.sgml
@@ -1645,6 +1645,23 @@ CREATE RULE shoelace_ins AS ON INSERT TO shoelace
<literal>RETURNING</literal> clause is simply ignored for <command>INSERT</command>.
</para>
+ <para>
+ Note that in the <literal>RETURNING</literal> clause of a rule,
+ <literal>OLD</literal> and <literal>NEW</literal> refer to the
+ pseudorelations added as extra range table entries to the rewritten
+ query, rather than old/new rows in the result relation. Thus, for
+ example, in a rule supporting <command>UPDATE</command> queries on this
+ view, if the <literal>RETURNING</literal> clause contained
+ <literal>old.sl_name</literal>, the old name would always be returned,
+ regardless of whether the <literal>RETURNING</literal> clause in the
+ query on the view specified <literal>OLD</literal> or <literal>NEW</literal>,
+ which might be confusing. To avoid this confusion, and support returning
+ old and new values in queries on the view, the <literal>RETURNING</literal>
+ clause in the rule definition should refer to entries from the result
+ relation such as <literal>shoelace_data.sl_name</literal>, without
+ specifying <literal>OLD</literal> or <literal>NEW</literal>.
+ </para>
+
<para>
Now assume that once in a while, a pack of shoelaces arrives at
the shop and a big parts list along with it. But you don't want
diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c
index 89514f7a4f..d8b92e6fc4 100644
--- a/src/backend/executor/execExpr.c
+++ b/src/backend/executor/execExpr.c
@@ -55,10 +55,15 @@
typedef struct ExprSetupInfo
{
- /* Highest attribute numbers fetched from inner/outer/scan tuple slots: */
+ /*
+ * Highest attribute numbers fetched from inner/outer/scan/old/new tuple
+ * slots:
+ */
AttrNumber last_inner;
AttrNumber last_outer;
AttrNumber last_scan;
+ AttrNumber last_old;
+ AttrNumber last_new;
/* MULTIEXPR SubPlan nodes appearing in the expression: */
List *multiexpr_subplans;
} ExprSetupInfo;
@@ -446,8 +451,25 @@ ExecBuildProjectionInfo(List *targetList,
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+
+ /*
+ * Get the tuple from the relation being scanned, or the
+ * old/new tuple slot, if old/new values were requested.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ scratch.opcode = EEOP_ASSIGN_SCAN_VAR;
+ break;
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_ASSIGN_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_ASSIGN_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ }
break;
}
@@ -535,7 +557,7 @@ ExecBuildUpdateProjection(List *targetList,
int nAssignableCols;
bool sawJunk;
Bitmapset *assignedCols;
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
ExprEvalStep scratch = {0};
int outerattnum;
ListCell *lc,
@@ -924,6 +946,7 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* system column */
scratch.d.var.attnum = variable->varattno;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -936,7 +959,20 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_SYSVAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ scratch.opcode = EEOP_SCAN_SYSVAR;
+ break;
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_SYSVAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ }
break;
}
}
@@ -945,6 +981,7 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* regular user column */
scratch.d.var.attnum = variable->varattno - 1;
scratch.d.var.vartype = variable->vartype;
+ scratch.d.var.varreturningtype = variable->varreturningtype;
switch (variable->varno)
{
case INNER_VAR:
@@ -957,7 +994,20 @@ ExecInitExprRec(Expr *node, ExprState *state,
/* INDEX_VAR is handled by default case */
default:
- scratch.opcode = EEOP_SCAN_VAR;
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ scratch.opcode = EEOP_SCAN_VAR;
+ break;
+ case VAR_RETURNING_OLD:
+ scratch.opcode = EEOP_OLD_VAR;
+ state->flags |= EEO_FLAG_HAS_OLD;
+ break;
+ case VAR_RETURNING_NEW:
+ scratch.opcode = EEOP_NEW_VAR;
+ state->flags |= EEO_FLAG_HAS_NEW;
+ break;
+ }
break;
}
}
@@ -2575,6 +2625,28 @@ ExecInitExprRec(Expr *node, ExprState *state,
break;
}
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ int retstep;
+
+ /* Skip expression evaluation if OLD/NEW row doesn't exist */
+ scratch.opcode = EEOP_RETURNINGEXPR;
+ scratch.d.returningexpr.nullflag = rexpr->retold ?
+ EEO_FLAG_OLD_IS_NULL : EEO_FLAG_NEW_IS_NULL;
+ scratch.d.returningexpr.jumpdone = -1; /* set below */
+ ExprEvalPushStep(state, &scratch);
+ retstep = state->steps_len - 1;
+
+ /* Steps to evaluate expression to return */
+ ExecInitExprRec(rexpr->retexpr, state, resv, resnull);
+
+ /* Jump target used if OLD/NEW row doesn't exist */
+ state->steps[retstep].d.returningexpr.jumpdone = state->steps_len;
+
+ break;
+ }
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(node));
@@ -2786,7 +2858,7 @@ ExecInitSubPlanExpr(SubPlan *subplan,
static void
ExecCreateExprSetupSteps(ExprState *state, Node *node)
{
- ExprSetupInfo info = {0, 0, 0, NIL};
+ ExprSetupInfo info = {0, 0, 0, 0, 0, NIL};
/* Prescan to find out what we need. */
expr_setup_walker(node, &info);
@@ -2809,8 +2881,8 @@ ExecPushExprSetupSteps(ExprState *state, ExprSetupInfo *info)
scratch.resnull = NULL;
/*
- * Add steps deforming the ExprState's inner/outer/scan slots as much as
- * required by any Vars appearing in the expression.
+ * Add steps deforming the ExprState's inner/outer/scan/old/new slots as
+ * much as required by any Vars appearing in the expression.
*/
if (info->last_inner > 0)
{
@@ -2842,6 +2914,26 @@ ExecPushExprSetupSteps(ExprState *state, ExprSetupInfo *info)
if (ExecComputeSlotInfo(state, &scratch))
ExprEvalPushStep(state, &scratch);
}
+ if (info->last_old > 0)
+ {
+ scratch.opcode = EEOP_OLD_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_old;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
+ if (info->last_new > 0)
+ {
+ scratch.opcode = EEOP_NEW_FETCHSOME;
+ scratch.d.fetch.last_var = info->last_new;
+ scratch.d.fetch.fixed = false;
+ scratch.d.fetch.kind = NULL;
+ scratch.d.fetch.known_desc = NULL;
+ if (ExecComputeSlotInfo(state, &scratch))
+ ExprEvalPushStep(state, &scratch);
+ }
/*
* Add steps to execute any MULTIEXPR SubPlans appearing in the
@@ -2888,7 +2980,18 @@ expr_setup_walker(Node *node, ExprSetupInfo *info)
/* INDEX_VAR is handled by default case */
default:
- info->last_scan = Max(info->last_scan, attnum);
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ info->last_scan = Max(info->last_scan, attnum);
+ break;
+ case VAR_RETURNING_OLD:
+ info->last_old = Max(info->last_old, attnum);
+ break;
+ case VAR_RETURNING_NEW:
+ info->last_new = Max(info->last_new, attnum);
+ break;
+ }
break;
}
return false;
@@ -2926,6 +3029,11 @@ expr_setup_walker(Node *node, ExprSetupInfo *info)
* evaluation of the expression will have the same type of slot, with an
* equivalent descriptor.
*
+ * EEOP_OLD_FETCHSOME and EEOP_NEW_FETCHSOME are used to process RETURNING, if
+ * OLD/NEW columns are referred to explicitly. In both cases, the tuple
+ * descriptor comes from the parent scan node, so we treat them the same as
+ * EEOP_SCAN_FETCHSOME.
+ *
* Returns true if the deforming step is required, false otherwise.
*/
static bool
@@ -2939,7 +3047,9 @@ ExecComputeSlotInfo(ExprState *state, ExprEvalStep *op)
Assert(opcode == EEOP_INNER_FETCHSOME ||
opcode == EEOP_OUTER_FETCHSOME ||
- opcode == EEOP_SCAN_FETCHSOME);
+ opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME);
if (op->d.fetch.known_desc != NULL)
{
@@ -2991,7 +3101,9 @@ ExecComputeSlotInfo(ExprState *state, ExprEvalStep *op)
desc = ExecGetResultType(os);
}
}
- else if (opcode == EEOP_SCAN_FETCHSOME)
+ else if (opcode == EEOP_SCAN_FETCHSOME ||
+ opcode == EEOP_OLD_FETCHSOME ||
+ opcode == EEOP_NEW_FETCHSOME)
{
desc = parent->scandesc;
@@ -3039,6 +3151,12 @@ ExecInitWholeRowVar(ExprEvalStep *scratch, Var *variable, ExprState *state)
scratch->d.wholerow.tupdesc = NULL; /* filled at runtime */
scratch->d.wholerow.junkFilter = NULL;
+ /* update ExprState flags if Var refers to OLD/NEW */
+ if (variable->varreturningtype == VAR_RETURNING_OLD)
+ state->flags |= EEO_FLAG_HAS_OLD;
+ else if (variable->varreturningtype == VAR_RETURNING_NEW)
+ state->flags |= EEO_FLAG_HAS_NEW;
+
/*
* If the input tuple came from a subquery, it might contain "resjunk"
* columns (such as GROUP BY or ORDER BY columns), which we don't want to
@@ -3541,7 +3659,7 @@ ExecBuildAggTrans(AggState *aggstate, AggStatePerPhase phase,
PlanState *parent = &aggstate->ss.ps;
ExprEvalStep scratch = {0};
bool isCombine = DO_AGGSPLIT_COMBINE(aggstate->aggsplit);
- ExprSetupInfo deform = {0, 0, 0, NIL};
+ ExprSetupInfo deform = {0, 0, 0, 0, 0, NIL};
state->expr = (Expr *) aggstate;
state->parent = parent;
@@ -4082,6 +4200,7 @@ ExecBuildHash32FromAttrs(TupleDesc desc, const TupleTableSlotOps *ops,
scratch.resnull = &fcinfo->args[0].isnull;
scratch.d.var.attnum = attnum;
scratch.d.var.vartype = TupleDescAttr(desc, attnum)->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
ExprEvalPushStep(state, &scratch);
@@ -4407,6 +4526,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc, TupleDesc rdesc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = latt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4415,6 +4535,7 @@ ExecBuildGroupingEqual(TupleDesc ldesc, TupleDesc rdesc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno - 1;
scratch.d.var.vartype = ratt->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4541,6 +4662,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_INNER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[0].value;
scratch.resnull = &fcinfo->args[0].isnull;
ExprEvalPushStep(state, &scratch);
@@ -4549,6 +4671,7 @@ ExecBuildParamSetEqual(TupleDesc desc,
scratch.opcode = EEOP_OUTER_VAR;
scratch.d.var.attnum = attno;
scratch.d.var.vartype = att->atttypid;
+ scratch.d.var.varreturningtype = VAR_RETURNING_DEFAULT;
scratch.resvalue = &fcinfo->args[1].value;
scratch.resnull = &fcinfo->args[1].isnull;
ExprEvalPushStep(state, &scratch);
diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c
index b2c00a0a1b..a99ba32044 100644
--- a/src/backend/executor/execExprInterp.c
+++ b/src/backend/executor/execExprInterp.c
@@ -462,6 +462,8 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
/*
* This array has to be in the same order as enum ExprEvalOp.
@@ -472,16 +474,24 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
&&CASE_EEOP_INNER_FETCHSOME,
&&CASE_EEOP_OUTER_FETCHSOME,
&&CASE_EEOP_SCAN_FETCHSOME,
+ &&CASE_EEOP_OLD_FETCHSOME,
+ &&CASE_EEOP_NEW_FETCHSOME,
&&CASE_EEOP_INNER_VAR,
&&CASE_EEOP_OUTER_VAR,
&&CASE_EEOP_SCAN_VAR,
+ &&CASE_EEOP_OLD_VAR,
+ &&CASE_EEOP_NEW_VAR,
&&CASE_EEOP_INNER_SYSVAR,
&&CASE_EEOP_OUTER_SYSVAR,
&&CASE_EEOP_SCAN_SYSVAR,
+ &&CASE_EEOP_OLD_SYSVAR,
+ &&CASE_EEOP_NEW_SYSVAR,
&&CASE_EEOP_WHOLEROW,
&&CASE_EEOP_ASSIGN_INNER_VAR,
&&CASE_EEOP_ASSIGN_OUTER_VAR,
&&CASE_EEOP_ASSIGN_SCAN_VAR,
+ &&CASE_EEOP_ASSIGN_OLD_VAR,
+ &&CASE_EEOP_ASSIGN_NEW_VAR,
&&CASE_EEOP_ASSIGN_TMP,
&&CASE_EEOP_ASSIGN_TMP_MAKE_RO,
&&CASE_EEOP_CONST,
@@ -523,6 +533,7 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
&&CASE_EEOP_SQLVALUEFUNCTION,
&&CASE_EEOP_CURRENTOFEXPR,
&&CASE_EEOP_NEXTVALUEEXPR,
+ &&CASE_EEOP_RETURNINGEXPR,
&&CASE_EEOP_ARRAYEXPR,
&&CASE_EEOP_ARRAYCOERCE,
&&CASE_EEOP_ROW,
@@ -591,6 +602,8 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
#if defined(EEO_USE_COMPUTED_GOTO)
EEO_DISPATCH();
@@ -630,6 +643,24 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, oldslot);
+
+ slot_getsomeattrs(oldslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_FETCHSOME)
+ {
+ CheckOpSlotCompatibility(op, newslot);
+
+ slot_getsomeattrs(newslot, op->d.fetch.last_var);
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_VAR)
{
int attnum = op->d.var.attnum;
@@ -673,6 +704,32 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ *op->resvalue = oldslot->tts_values[attnum];
+ *op->resnull = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_VAR)
+ {
+ int attnum = op->d.var.attnum;
+
+ /* See EEOP_INNER_VAR comments */
+
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ *op->resvalue = newslot->tts_values[attnum];
+ *op->resnull = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_INNER_SYSVAR)
{
ExecEvalSysVar(state, op, econtext, innerslot);
@@ -691,6 +748,18 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_OLD_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, oldslot);
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_NEW_SYSVAR)
+ {
+ ExecEvalSysVar(state, op, econtext, newslot);
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_WHOLEROW)
{
/* too complex for an inline implementation */
@@ -750,6 +819,40 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_ASSIGN_OLD_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < oldslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = oldslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = oldslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
+ EEO_CASE(EEOP_ASSIGN_NEW_VAR)
+ {
+ int resultnum = op->d.assign_var.resultnum;
+ int attnum = op->d.assign_var.attnum;
+
+ /*
+ * We do not need CheckVarSlotCompatibility here; that was taken
+ * care of at compilation time. But see EEOP_INNER_VAR comments.
+ */
+ Assert(attnum >= 0 && attnum < newslot->tts_nvalid);
+ Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
+ resultslot->tts_values[resultnum] = newslot->tts_values[attnum];
+ resultslot->tts_isnull[resultnum] = newslot->tts_isnull[attnum];
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ASSIGN_TMP)
{
int resultnum = op->d.assign_tmp.resultnum;
@@ -1438,6 +1541,23 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull)
EEO_NEXT();
}
+ EEO_CASE(EEOP_RETURNINGEXPR)
+ {
+ /*
+ * The next op actually evaluates the expression. If the OLD/NEW
+ * row doesn't exist, skip that and return NULL.
+ */
+ if (state->flags & op->d.returningexpr.nullflag)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+
+ EEO_JUMP(op->d.returningexpr.jumpdone);
+ }
+
+ EEO_NEXT();
+ }
+
EEO_CASE(EEOP_ARRAYEXPR)
{
/* too complex for an inline implementation */
@@ -2119,10 +2239,14 @@ CheckExprStillValid(ExprState *state, ExprContext *econtext)
TupleTableSlot *innerslot;
TupleTableSlot *outerslot;
TupleTableSlot *scanslot;
+ TupleTableSlot *oldslot;
+ TupleTableSlot *newslot;
innerslot = econtext->ecxt_innertuple;
outerslot = econtext->ecxt_outertuple;
scanslot = econtext->ecxt_scantuple;
+ oldslot = econtext->ecxt_oldtuple;
+ newslot = econtext->ecxt_newtuple;
for (int i = 0; i < state->steps_len; i++)
{
@@ -2153,6 +2277,22 @@ CheckExprStillValid(ExprState *state, ExprContext *econtext)
CheckVarSlotCompatibility(scanslot, attnum + 1, op->d.var.vartype);
break;
}
+
+ case EEOP_OLD_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(oldslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
+
+ case EEOP_NEW_VAR:
+ {
+ int attnum = op->d.var.attnum;
+
+ CheckVarSlotCompatibility(newslot, attnum + 1, op->d.var.vartype);
+ break;
+ }
default:
break;
}
@@ -5113,7 +5253,7 @@ void
ExecEvalWholeRowVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext)
{
Var *variable = op->d.wholerow.var;
- TupleTableSlot *slot;
+ TupleTableSlot *slot = NULL;
TupleDesc output_tupdesc;
MemoryContext oldcontext;
HeapTupleHeader dtuple;
@@ -5138,8 +5278,40 @@ ExecEvalWholeRowVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext)
/* INDEX_VAR is handled by default case */
default:
- /* get the tuple from the relation being scanned */
- slot = econtext->ecxt_scantuple;
+
+ /*
+ * Get the tuple from the relation being scanned.
+ *
+ * By default, this uses the "scan" tuple slot, but a wholerow Var
+ * in the RETURNING list may explicitly refer to OLD/NEW. If the
+ * OLD/NEW row doesn't exist, we just return NULL.
+ */
+ switch (variable->varreturningtype)
+ {
+ case VAR_RETURNING_DEFAULT:
+ slot = econtext->ecxt_scantuple;
+ break;
+
+ case VAR_RETURNING_OLD:
+ if (state->flags & EEO_FLAG_OLD_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_oldtuple;
+ break;
+
+ case VAR_RETURNING_NEW:
+ if (state->flags & EEO_FLAG_NEW_IS_NULL)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ slot = econtext->ecxt_newtuple;
+ break;
+ }
break;
}
@@ -5342,6 +5514,27 @@ ExecEvalSysVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext,
{
Datum d;
+ /*
+ * For OLD/NEW system attributes, check whether the OLD/NEW row exists. If
+ * it doesn't, the OLD/NEW system attribute is NULL.
+ */
+ if (op->d.var.varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ bool rowIsNull;
+
+ if (op->d.var.varreturningtype == VAR_RETURNING_OLD)
+ rowIsNull = (state->flags & EEO_FLAG_OLD_IS_NULL) != 0;
+ else
+ rowIsNull = (state->flags & EEO_FLAG_NEW_IS_NULL) != 0;
+
+ if (rowIsNull)
+ {
+ *op->resvalue = (Datum) 0;
+ *op->resnull = true;
+ return;
+ }
+ }
+
/* slot_getsysattr has sufficient defenses against bad attnums */
d = slot_getsysattr(slot,
op->d.var.attnum,
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
index a06295b6ba..86a81cb0ff 100644
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -1257,6 +1257,7 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo,
resultRelInfo->ri_ReturningSlot = NULL;
resultRelInfo->ri_TrigOldSlot = NULL;
resultRelInfo->ri_TrigNewSlot = NULL;
+ resultRelInfo->ri_AllNullSlot = NULL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_MATCHED] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_SOURCE] = NIL;
resultRelInfo->ri_MergeActions[MERGE_WHEN_NOT_MATCHED_BY_TARGET] = NIL;
diff --git a/src/backend/executor/execUtils.c b/src/backend/executor/execUtils.c
index f71899463b..7c539de5cf 100644
--- a/src/backend/executor/execUtils.c
+++ b/src/backend/executor/execUtils.c
@@ -1242,6 +1242,34 @@ ExecGetReturningSlot(EState *estate, ResultRelInfo *relInfo)
return relInfo->ri_ReturningSlot;
}
+/*
+ * Return a relInfo's all-NULL tuple slot for processing returning tuples.
+ *
+ * Note: this slot is intentionally filled with NULLs in every column, and
+ * should be considered read-only --- the caller must not update it.
+ */
+TupleTableSlot *
+ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo)
+{
+ if (relInfo->ri_AllNullSlot == NULL)
+ {
+ Relation rel = relInfo->ri_RelationDesc;
+ MemoryContext oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
+ TupleTableSlot *slot;
+
+ slot = ExecInitExtraTupleSlot(estate,
+ RelationGetDescr(rel),
+ table_slot_callbacks(rel));
+ ExecStoreAllNullTuple(slot);
+
+ relInfo->ri_AllNullSlot = slot;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ return relInfo->ri_AllNullSlot;
+}
+
/*
* Return the map needed to convert given child result relation's tuples to
* the rowtype of the query's main target ("root") relation. Note that a
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 1af8c9caf6..bc82e035ba 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -101,6 +101,13 @@ typedef struct ModifyTableContext
*/
TM_FailureData tmfd;
+ /*
+ * The tuple deleted when doing a cross-partition UPDATE with a RETURNING
+ * clause that refers to OLD columns (converted to the root's tuple
+ * descriptor).
+ */
+ TupleTableSlot *cpDeletedSlot;
+
/*
* The tuple projected by the INSERT's RETURNING clause, when doing a
* cross-partition UPDATE
@@ -243,34 +250,81 @@ ExecCheckPlanOutput(Relation resultRel, List *targetList)
/*
* ExecProcessReturning --- evaluate a RETURNING list
*
+ * context: context for the ModifyTable operation
* resultRelInfo: current result rel
- * tupleSlot: slot holding tuple actually inserted/updated/deleted
+ * cmdType: operation/merge action performed (INSERT, UPDATE, or DELETE)
+ * oldSlot: slot holding old tuple deleted or updated
+ * newSlot: slot holding new tuple inserted or updated
* planSlot: slot holding tuple returned by top subplan node
*
- * Note: If tupleSlot is NULL, the FDW should have already provided econtext's
- * scan tuple.
+ * Note: If oldSlot and newSlot are NULL, the FDW should have already provided
+ * econtext's scan tuple and its old & new tuples are not needed (FDW direct-
+ * modify is disabled if the RETURNING list refers to any OLD/NEW values).
*
* Returns a slot holding the result tuple
*/
static TupleTableSlot *
-ExecProcessReturning(ResultRelInfo *resultRelInfo,
- TupleTableSlot *tupleSlot,
+ExecProcessReturning(ModifyTableContext *context,
+ ResultRelInfo *resultRelInfo,
+ CmdType cmdType,
+ TupleTableSlot *oldSlot,
+ TupleTableSlot *newSlot,
TupleTableSlot *planSlot)
{
+ EState *estate = context->estate;
ProjectionInfo *projectReturning = resultRelInfo->ri_projectReturning;
ExprContext *econtext = projectReturning->pi_exprContext;
/* Make tuple and any needed join variables available to ExecProject */
- if (tupleSlot)
- econtext->ecxt_scantuple = tupleSlot;
+ switch (cmdType)
+ {
+ case CMD_INSERT:
+ case CMD_UPDATE:
+ /* return new tuple by default */
+ if (newSlot)
+ econtext->ecxt_scantuple = newSlot;
+ break;
+
+ case CMD_DELETE:
+ /* return old tuple by default */
+ if (oldSlot)
+ econtext->ecxt_scantuple = oldSlot;
+ break;
+
+ default:
+ elog(ERROR, "unrecognized commandType: %d", (int) cmdType);
+ }
econtext->ecxt_outertuple = planSlot;
+ /* Make old/new tuples available to ExecProject, if required */
+ if (oldSlot)
+ econtext->ecxt_oldtuple = oldSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ econtext->ecxt_oldtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_oldtuple = NULL; /* No references to OLD columns */
+
+ if (newSlot)
+ econtext->ecxt_newtuple = newSlot;
+ else if (projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW)
+ econtext->ecxt_newtuple = ExecGetAllNullSlot(estate, resultRelInfo);
+ else
+ econtext->ecxt_newtuple = NULL; /* No references to NEW columns */
+
/*
- * RETURNING expressions might reference the tableoid column, so
- * reinitialize tts_tableOid before evaluating them.
+ * Tell ExecProject whether or not the OLD/NEW rows actually exist. This
+ * information is required to evaluate ReturningExpr nodes and also in
+ * ExecEvalSysVar() and ExecEvalWholeRowVar().
*/
- econtext->ecxt_scantuple->tts_tableOid =
- RelationGetRelid(resultRelInfo->ri_RelationDesc);
+ if (oldSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_OLD_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_OLD_IS_NULL;
+
+ if (newSlot == NULL)
+ projectReturning->pi_state.flags |= EEO_FLAG_NEW_IS_NULL;
+ else
+ projectReturning->pi_state.flags &= ~EEO_FLAG_NEW_IS_NULL;
/* Compute the RETURNING expressions */
return ExecProject(projectReturning);
@@ -1204,7 +1258,56 @@ ExecInsert(ModifyTableContext *context,
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- result = ExecProcessReturning(resultRelInfo, slot, planSlot);
+ {
+ TupleTableSlot *oldSlot = NULL;
+
+ /*
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD columns, ExecDelete() will have saved the tuple
+ * deleted from the original partition, which we must use here to
+ * compute the OLD column values. Otherwise, all OLD column values
+ * will be NULL.
+ */
+ if (context->cpDeletedSlot)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the OLD tuple to the new partition's format/slot, if
+ * needed. Note that ExceDelete() already converted it to the
+ * root's partition's format/slot.
+ */
+ oldSlot = context->cpDeletedSlot;
+ tupconv_map = ExecGetRootToChildMap(resultRelInfo, estate);
+ if (tupconv_map != NULL)
+ {
+ oldSlot = execute_attr_map_slot(tupconv_map->attrMap,
+ oldSlot,
+ ExecGetReturningSlot(estate,
+ resultRelInfo));
+
+ oldSlot->tts_tableOid = context->cpDeletedSlot->tts_tableOid;
+ ItemPointerCopy(&context->cpDeletedSlot->tts_tid, &oldSlot->tts_tid);
+ }
+ }
+
+ result = ExecProcessReturning(context, resultRelInfo, CMD_INSERT,
+ oldSlot, slot, planSlot);
+
+ /*
+ * For a cross-partition UPDATE, release the old tuple, first making
+ * sure that the result slot has a local copy of any pass-by-reference
+ * values.
+ */
+ if (context->cpDeletedSlot)
+ {
+ ExecMaterializeSlot(result);
+ ExecClearTuple(oldSlot);
+ if (context->cpDeletedSlot != oldSlot)
+ ExecClearTuple(context->cpDeletedSlot);
+ context->cpDeletedSlot = NULL;
+ }
+ }
if (inserted_tuple)
*inserted_tuple = slot;
@@ -1442,6 +1545,7 @@ ExecDelete(ModifyTableContext *context,
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
TupleTableSlot *slot = NULL;
TM_Result result;
+ bool saveOld;
if (tupleDeleted)
*tupleDeleted = false;
@@ -1676,8 +1780,17 @@ ldelete:
ExecDeleteEpilogue(context, resultRelInfo, tupleid, oldtuple, changingPart);
- /* Process RETURNING if present and if requested */
- if (processReturning && resultRelInfo->ri_projectReturning)
+ /*
+ * Process RETURNING if present and if requested.
+ *
+ * If this is part of a cross-partition UPDATE, and the RETURNING list
+ * refers to any OLD column values, save the old tuple here for later
+ * processing of the RETURNING list by ExecInsert().
+ */
+ saveOld = changingPart && resultRelInfo->ri_projectReturning &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD;
+
+ if (resultRelInfo->ri_projectReturning && (processReturning || saveOld))
{
/*
* We have to put the target tuple into a slot, which means first we
@@ -1705,7 +1818,41 @@ ldelete:
}
}
- rslot = ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ /*
+ * If required, save the old tuple for later processing of the
+ * RETURNING list by ExecInsert().
+ */
+ if (saveOld)
+ {
+ TupleConversionMap *tupconv_map;
+
+ /*
+ * Convert the tuple into the root partition's format/slot, if
+ * needed. ExecInsert() will then convert it to the new
+ * partition's format/slot, if necessary.
+ */
+ tupconv_map = ExecGetChildToRootMap(resultRelInfo);
+ if (tupconv_map != NULL)
+ {
+ ResultRelInfo *rootRelInfo = context->mtstate->rootResultRelInfo;
+ TupleTableSlot *oldSlot = slot;
+
+ slot = execute_attr_map_slot(tupconv_map->attrMap,
+ slot,
+ ExecGetReturningSlot(estate,
+ rootRelInfo));
+
+ slot->tts_tableOid = oldSlot->tts_tableOid;
+ ItemPointerCopy(&oldSlot->tts_tid, &slot->tts_tid);
+ }
+
+ context->cpDeletedSlot = slot;
+
+ return NULL;
+ }
+
+ rslot = ExecProcessReturning(context, resultRelInfo, CMD_DELETE,
+ slot, NULL, context->planSlot);
/*
* Before releasing the target tuple again, make sure rslot has a
@@ -1758,6 +1905,7 @@ ExecCrossPartitionUpdate(ModifyTableContext *context,
bool tuple_deleted;
TupleTableSlot *epqslot = NULL;
+ context->cpDeletedSlot = NULL;
context->cpUpdateReturningSlot = NULL;
*retry_slot = NULL;
@@ -2258,6 +2406,7 @@ ExecCrossPartitionUpdateForeignKey(ModifyTableContext *context,
* the planSlot. oldtuple is passed to foreign table triggers; it is
* NULL when the foreign table has no relevant triggers.
*
+ * oldSlot contains the old tuple value.
* slot contains the new tuple value to be stored.
* planSlot is the output of the ModifyTable's subplan; we use it
* to access values from other input tables (for RETURNING),
@@ -2270,8 +2419,8 @@ ExecCrossPartitionUpdateForeignKey(ModifyTableContext *context,
*/
static TupleTableSlot *
ExecUpdate(ModifyTableContext *context, ResultRelInfo *resultRelInfo,
- ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot,
- bool canSetTag)
+ ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *oldSlot,
+ TupleTableSlot *slot, bool canSetTag)
{
EState *estate = context->estate;
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
@@ -2389,7 +2538,6 @@ redo_act:
{
TupleTableSlot *inputslot;
TupleTableSlot *epqslot;
- TupleTableSlot *oldSlot;
if (IsolationUsesXactSnapshot())
ereport(ERROR,
@@ -2504,7 +2652,8 @@ redo_act:
/* Process RETURNING if present */
if (resultRelInfo->ri_projectReturning)
- return ExecProcessReturning(resultRelInfo, slot, context->planSlot);
+ return ExecProcessReturning(context, resultRelInfo, CMD_UPDATE,
+ oldSlot, slot, context->planSlot);
return NULL;
}
@@ -2724,16 +2873,23 @@ ExecOnConflictUpdate(ModifyTableContext *context,
/* Execute UPDATE with projection */
*returning = ExecUpdate(context, resultRelInfo,
- conflictTid, NULL,
+ conflictTid, NULL, existing,
resultRelInfo->ri_onConflict->oc_ProjSlot,
canSetTag);
/*
* Clear out existing tuple, as there might not be another conflict among
* the next input rows. Don't want to hold resources till the end of the
- * query.
+ * query. First though, make sure that the returning slot, if any, has a
+ * local copy of any OLD pass-by-reference values, if it refers to any OLD
+ * columns.
*/
+ if (*returning != NULL &&
+ resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD)
+ ExecMaterializeSlot(*returning);
+
ExecClearTuple(existing);
+
return true;
}
@@ -3338,13 +3494,20 @@ lmerge_matched:
switch (commandType)
{
case CMD_UPDATE:
- rslot = ExecProcessReturning(resultRelInfo, newslot,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_UPDATE,
+ resultRelInfo->ri_oldTupleSlot,
+ newslot,
context->planSlot);
break;
case CMD_DELETE:
- rslot = ExecProcessReturning(resultRelInfo,
+ rslot = ExecProcessReturning(context,
+ resultRelInfo,
+ CMD_DELETE,
resultRelInfo->ri_oldTupleSlot,
+ NULL,
context->planSlot);
break;
@@ -3894,6 +4057,7 @@ ExecModifyTable(PlanState *pstate)
if (node->mt_merge_pending_not_matched != NULL)
{
context.planSlot = node->mt_merge_pending_not_matched;
+ context.cpDeletedSlot = NULL;
slot = ExecMergeNotMatched(&context, node->resultRelInfo,
node->canSetTag);
@@ -3913,6 +4077,7 @@ ExecModifyTable(PlanState *pstate)
/* Fetch the next row from subplan */
context.planSlot = ExecProcNode(subplanstate);
+ context.cpDeletedSlot = NULL;
/* No more tuples to process? */
if (TupIsNull(context.planSlot))
@@ -3980,9 +4145,15 @@ ExecModifyTable(PlanState *pstate)
* A scan slot containing the data that was actually inserted,
* updated or deleted has already been made available to
* ExecProcessReturning by IterateDirectModify, so no need to
- * provide it here.
+ * provide it here. The individual old and new slots are not
+ * needed, since direct-modify is disabled if the RETURNING list
+ * refers to OLD/NEW values.
*/
- slot = ExecProcessReturning(resultRelInfo, NULL, context.planSlot);
+ Assert((resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_OLD) == 0 &&
+ (resultRelInfo->ri_projectReturning->pi_state.flags & EEO_FLAG_HAS_NEW) == 0);
+
+ slot = ExecProcessReturning(&context, resultRelInfo, operation,
+ NULL, NULL, context.planSlot);
return slot;
}
@@ -4172,7 +4343,7 @@ ExecModifyTable(PlanState *pstate)
/* Now apply the update. */
slot = ExecUpdate(&context, resultRelInfo, tupleid, oldtuple,
- slot, node->canSetTag);
+ oldSlot, slot, node->canSetTag);
if (tuplock)
UnlockTuple(resultRelInfo->ri_RelationDesc, tupleid,
InplaceUpdateTupleLock);
diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c
index 6c4915e373..6b8f4b1971 100644
--- a/src/backend/jit/llvm/llvmjit_expr.c
+++ b/src/backend/jit/llvm/llvmjit_expr.c
@@ -105,6 +105,8 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_innerslot;
LLVMValueRef v_outerslot;
LLVMValueRef v_scanslot;
+ LLVMValueRef v_oldslot;
+ LLVMValueRef v_newslot;
LLVMValueRef v_resultslot;
/* nulls/values of slots */
@@ -114,6 +116,10 @@ llvm_compile_expr(ExprState *state)
LLVMValueRef v_outernulls;
LLVMValueRef v_scanvalues;
LLVMValueRef v_scannulls;
+ LLVMValueRef v_oldvalues;
+ LLVMValueRef v_oldnulls;
+ LLVMValueRef v_newvalues;
+ LLVMValueRef v_newnulls;
LLVMValueRef v_resultvalues;
LLVMValueRef v_resultnulls;
@@ -200,6 +206,16 @@ llvm_compile_expr(ExprState *state)
v_econtext,
FIELDNO_EXPRCONTEXT_OUTERTUPLE,
"v_outerslot");
+ v_oldslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_OLDTUPLE,
+ "v_oldslot");
+ v_newslot = l_load_struct_gep(b,
+ StructExprContext,
+ v_econtext,
+ FIELDNO_EXPRCONTEXT_NEWTUPLE,
+ "v_newslot");
v_resultslot = l_load_struct_gep(b,
StructExprState,
v_state,
@@ -237,6 +253,26 @@ llvm_compile_expr(ExprState *state)
v_outerslot,
FIELDNO_TUPLETABLESLOT_ISNULL,
"v_outernulls");
+ v_oldvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_oldvalues");
+ v_oldnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_oldslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_oldnulls");
+ v_newvalues = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_VALUES,
+ "v_newvalues");
+ v_newnulls = l_load_struct_gep(b,
+ StructTupleTableSlot,
+ v_newslot,
+ FIELDNO_TUPLETABLESLOT_ISNULL,
+ "v_newnulls");
v_resultvalues = l_load_struct_gep(b,
StructTupleTableSlot,
v_resultslot,
@@ -302,6 +338,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_FETCHSOME:
case EEOP_OUTER_FETCHSOME:
case EEOP_SCAN_FETCHSOME:
+ case EEOP_OLD_FETCHSOME:
+ case EEOP_NEW_FETCHSOME:
{
TupleDesc desc = NULL;
LLVMValueRef v_slot;
@@ -326,8 +364,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_FETCHSOME)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_FETCHSOME)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_FETCHSOME)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
/*
* Check if all required attributes are available, or
@@ -396,6 +438,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_VAR:
case EEOP_OUTER_VAR:
case EEOP_SCAN_VAR:
+ case EEOP_OLD_VAR:
+ case EEOP_NEW_VAR:
{
LLVMValueRef value,
isnull;
@@ -413,11 +457,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
v_attnum = l_int32_const(lc, op->d.var.attnum);
value = l_load_gep1(b, TypeSizeT, v_values, v_attnum, "");
@@ -432,6 +486,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_INNER_SYSVAR:
case EEOP_OUTER_SYSVAR:
case EEOP_SCAN_SYSVAR:
+ case EEOP_OLD_SYSVAR:
+ case EEOP_NEW_SYSVAR:
{
LLVMValueRef v_slot;
@@ -439,8 +495,12 @@ llvm_compile_expr(ExprState *state)
v_slot = v_innerslot;
else if (opcode == EEOP_OUTER_SYSVAR)
v_slot = v_outerslot;
- else
+ else if (opcode == EEOP_SCAN_SYSVAR)
v_slot = v_scanslot;
+ else if (opcode == EEOP_OLD_SYSVAR)
+ v_slot = v_oldslot;
+ else
+ v_slot = v_newslot;
build_EvalXFunc(b, mod, "ExecEvalSysVar",
v_state, op, v_econtext, v_slot);
@@ -458,6 +518,8 @@ llvm_compile_expr(ExprState *state)
case EEOP_ASSIGN_INNER_VAR:
case EEOP_ASSIGN_OUTER_VAR:
case EEOP_ASSIGN_SCAN_VAR:
+ case EEOP_ASSIGN_OLD_VAR:
+ case EEOP_ASSIGN_NEW_VAR:
{
LLVMValueRef v_value;
LLVMValueRef v_isnull;
@@ -478,11 +540,21 @@ llvm_compile_expr(ExprState *state)
v_values = v_outervalues;
v_nulls = v_outernulls;
}
- else
+ else if (opcode == EEOP_ASSIGN_SCAN_VAR)
{
v_values = v_scanvalues;
v_nulls = v_scannulls;
}
+ else if (opcode == EEOP_ASSIGN_OLD_VAR)
+ {
+ v_values = v_oldvalues;
+ v_nulls = v_oldnulls;
+ }
+ else
+ {
+ v_values = v_newvalues;
+ v_nulls = v_newnulls;
+ }
/* load data */
v_attnum = l_int32_const(lc, op->d.assign_var.attnum);
@@ -1654,6 +1726,45 @@ llvm_compile_expr(ExprState *state)
LLVMBuildBr(b, opblocks[opno + 1]);
break;
+ case EEOP_RETURNINGEXPR:
+ {
+ LLVMBasicBlockRef b_isnull;
+ LLVMValueRef v_flagsp;
+ LLVMValueRef v_flags;
+ LLVMValueRef v_nullflag;
+
+ b_isnull = l_bb_before_v(opblocks[opno + 1],
+ "op.%d.row.isnull", opno);
+
+ /*
+ * The next op actually evaluates the expression. If the
+ * OLD/NEW row doesn't exist, skip that and return NULL.
+ */
+ v_flagsp = l_struct_gep(b,
+ StructExprState,
+ v_state,
+ FIELDNO_EXPRSTATE_FLAGS,
+ "v.state.flags");
+ v_flags = l_load(b, TypeStorageBool, v_flagsp, "");
+
+ v_nullflag = l_int8_const(lc, op->d.returningexpr.nullflag);
+
+ LLVMBuildCondBr(b,
+ LLVMBuildICmp(b, LLVMIntEQ,
+ LLVMBuildAnd(b, v_flags,
+ v_nullflag, ""),
+ l_sbool_const(0), ""),
+ opblocks[opno + 1], b_isnull);
+
+ LLVMPositionBuilderAtEnd(b, b_isnull);
+
+ LLVMBuildStore(b, l_sizet_const(0), v_resvaluep);
+ LLVMBuildStore(b, l_sbool_const(1), v_resnullp);
+
+ LLVMBuildBr(b, opblocks[op->d.returningexpr.jumpdone]);
+ break;
+ }
+
case EEOP_ARRAYEXPR:
build_EvalXFunc(b, mod, "ExecEvalArrayExpr",
v_state, op);
diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c
index 6b66bc1828..0d32c7bcc0 100644
--- a/src/backend/nodes/makefuncs.c
+++ b/src/backend/nodes/makefuncs.c
@@ -80,12 +80,14 @@ makeVar(int varno,
var->varlevelsup = varlevelsup;
/*
- * Only a few callers need to make Var nodes with non-null varnullingrels,
- * or with varnosyn/varattnosyn different from varno/varattno. We don't
- * provide separate arguments for them, but just initialize them to NULL
- * and the given varno/varattno. This reduces code clutter and chance of
- * error for most callers.
+ * Only a few callers need to make Var nodes with varreturningtype
+ * different from VAR_RETURNING_DEFAULT, non-null varnullingrels, or with
+ * varnosyn/varattnosyn different from varno/varattno. We don't provide
+ * separate arguments for them, but just initialize them to sensible
+ * default values. This reduces code clutter and chance of error for most
+ * callers.
*/
+ var->varreturningtype = VAR_RETURNING_DEFAULT;
var->varnullingrels = NULL;
var->varnosyn = (Index) varno;
var->varattnosyn = varattno;
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index df779137c9..6eb53b7bd4 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -278,6 +278,9 @@ exprType(const Node *expr)
type = exprType((Node *) n->expr);
}
break;
+ case T_ReturningExpr:
+ type = exprType((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
type = exprType((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -529,6 +532,8 @@ exprTypmod(const Node *expr)
return ((const CoerceToDomainValue *) expr)->typeMod;
case T_SetToDefault:
return ((const SetToDefault *) expr)->typeMod;
+ case T_ReturningExpr:
+ return exprTypmod((Node *) ((const ReturningExpr *) expr)->retexpr);
case T_PlaceHolderVar:
return exprTypmod((Node *) ((const PlaceHolderVar *) expr)->phexpr);
default:
@@ -1047,6 +1052,9 @@ exprCollation(const Node *expr)
case T_InferenceElem:
coll = exprCollation((Node *) ((const InferenceElem *) expr)->expr);
break;
+ case T_ReturningExpr:
+ coll = exprCollation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_PlaceHolderVar:
coll = exprCollation((Node *) ((const PlaceHolderVar *) expr)->phexpr);
break;
@@ -1298,6 +1306,10 @@ exprSetCollation(Node *expr, Oid collation)
/* NextValueExpr's result is an integer type ... */
Assert(!OidIsValid(collation)); /* ... so never set a collation */
break;
+ case T_ReturningExpr:
+ exprSetCollation((Node *) ((ReturningExpr *) expr)->retexpr,
+ collation);
+ break;
default:
elog(ERROR, "unrecognized node type: %d", (int) nodeTag(expr));
break;
@@ -1624,6 +1636,9 @@ exprLocation(const Node *expr)
case T_SetToDefault:
loc = ((const SetToDefault *) expr)->location;
break;
+ case T_ReturningExpr:
+ loc = exprLocation((Node *) ((const ReturningExpr *) expr)->retexpr);
+ break;
case T_TargetEntry:
/* just use argument's location */
loc = exprLocation((Node *) ((const TargetEntry *) expr)->expr);
@@ -2613,6 +2628,8 @@ expression_tree_walker_impl(Node *node,
return WALK(((PlaceHolderVar *) node)->phexpr);
case T_InferenceElem:
return WALK(((InferenceElem *) node)->expr);
+ case T_ReturningExpr:
+ return WALK(((ReturningExpr *) node)->retexpr);
case T_AppendRelInfo:
{
AppendRelInfo *appinfo = (AppendRelInfo *) node;
@@ -3454,6 +3471,16 @@ expression_tree_mutator_impl(Node *node,
return (Node *) newnode;
}
break;
+ case T_ReturningExpr:
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+ ReturningExpr *newnode;
+
+ FLATCOPY(newnode, rexpr, ReturningExpr);
+ MUTATE(newnode->retexpr, rexpr->retexpr, Expr *);
+ return (Node *) newnode;
+ }
+ break;
case T_TargetEntry:
{
TargetEntry *targetentry = (TargetEntry *) node;
@@ -4005,6 +4032,7 @@ raw_expression_tree_walker_impl(Node *node,
case T_A_Const:
case T_A_Star:
case T_MergeSupportFunc:
+ case T_ReturningOption:
/* primitive node types with no subnodes */
break;
case T_Alias:
@@ -4233,7 +4261,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->onConflictClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4249,7 +4277,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->whereClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4267,7 +4295,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->fromClause))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4285,7 +4313,7 @@ raw_expression_tree_walker_impl(Node *node,
return true;
if (WALK(stmt->mergeWhenClauses))
return true;
- if (WALK(stmt->returningList))
+ if (WALK(stmt->returningClause))
return true;
if (WALK(stmt->withClause))
return true;
@@ -4303,6 +4331,16 @@ raw_expression_tree_walker_impl(Node *node,
return true;
}
break;
+ case T_ReturningClause:
+ {
+ ReturningClause *returning = (ReturningClause *) node;
+
+ if (WALK(returning->options))
+ return true;
+ if (WALK(returning->exprs))
+ return true;
+ }
+ break;
case T_SelectStmt:
{
SelectStmt *stmt = (SelectStmt *) node;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 3364589391..1115ebeee2 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -3985,6 +3985,7 @@ subquery_push_qual(Query *subquery, RangeTblEntry *rte, Index rti, Node *qual)
*/
qual = ReplaceVarsFromTargetList(qual, rti, 0, rte,
subquery->targetList,
+ subquery->resultRelation,
REPLACEVARS_REPORT_ERROR, 0,
&subquery->hasSubLinks);
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 1caad5f3a6..1106cd85f0 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -7121,6 +7121,8 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
int epqParam)
{
ModifyTable *node = makeNode(ModifyTable);
+ bool returning_old_or_new = false;
+ bool returning_old_or_new_valid = false;
List *fdw_private_list;
Bitmapset *direct_modify_plans;
ListCell *lc;
@@ -7185,6 +7187,8 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
}
node->updateColnosLists = updateColnosLists;
node->withCheckOptionLists = withCheckOptionLists;
+ node->returningOldAlias = root->parse->returningOldAlias;
+ node->returningNewAlias = root->parse->returningNewAlias;
node->returningLists = returningLists;
node->rowMarks = rowMarks;
node->mergeActionLists = mergeActionLists;
@@ -7265,7 +7269,8 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
* callback functions needed for that and (2) there are no local
* structures that need to be run for each modified row: row-level
* triggers on the foreign table, stored generated columns, WITH CHECK
- * OPTIONs from parent views.
+ * OPTIONs from parent views, or Vars returning OLD/NEW in the
+ * RETURNING list.
*/
direct_modify = false;
if (fdwroutine != NULL &&
@@ -7276,7 +7281,18 @@ make_modifytable(PlannerInfo *root, Plan *subplan,
withCheckOptionLists == NIL &&
!has_row_triggers(root, rti, operation) &&
!has_stored_generated_columns(root, rti))
- direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ {
+ /* returning_old_or_new is the same for all result relations */
+ if (!returning_old_or_new_valid)
+ {
+ returning_old_or_new =
+ contain_vars_returning_old_or_new((Node *)
+ root->parse->returningList);
+ returning_old_or_new_valid = true;
+ }
+ if (!returning_old_or_new)
+ direct_modify = fdwroutine->PlanDirectModify(root, node, rti, i);
+ }
if (direct_modify)
direct_modify_plans = bms_add_member(direct_modify_plans, i);
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 8136358912..fff2655595 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -3070,6 +3070,21 @@ fix_join_expr_mutator(Node *node, fix_join_expr_context *context)
{
Var *var = (Var *) node;
+ /*
+ * Verify that Vars with non-default varreturningtype only appear in
+ * the RETURNING list, and refer to the target relation.
+ */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ if (context->inner_itlist != NULL ||
+ context->outer_itlist == NULL ||
+ context->acceptable_rel == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+ if (var->varno != context->acceptable_rel)
+ elog(ERROR, "wrong varno %d (expected %d) for variable returning old/new",
+ var->varno, context->acceptable_rel);
+ }
+
/* Look for the var in the input tlists, first in the outer */
if (context->outer_itlist)
{
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index eaaf8c1b49..adf0d09134 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -354,17 +354,19 @@ build_subplan(PlannerInfo *root, Plan *plan, Path *path,
Node *arg = pitem->item;
/*
- * The Var, PlaceHolderVar, Aggref or GroupingFunc has already been
- * adjusted to have the correct varlevelsup, phlevelsup, or
- * agglevelsup.
+ * The Var, PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr has
+ * already been adjusted to have the correct varlevelsup, phlevelsup,
+ * agglevelsup, or retlevelsup.
*
- * If it's a PlaceHolderVar, Aggref or GroupingFunc, its arguments
- * might contain SubLinks, which have not yet been processed (see the
- * comments for SS_replace_correlation_vars). Do that now.
+ * If it's a PlaceHolderVar, Aggref, GroupingFunc or ReturningExpr,
+ * its arguments might contain SubLinks, which have not yet been
+ * processed (see the comments for SS_replace_correlation_vars). Do
+ * that now.
*/
if (IsA(arg, PlaceHolderVar) ||
IsA(arg, Aggref) ||
- IsA(arg, GroupingFunc))
+ IsA(arg, GroupingFunc) ||
+ IsA(arg, ReturningExpr))
arg = SS_process_sublinks(root, arg, false);
splan->parParam = lappend_int(splan->parParam, pitem->paramId);
@@ -1863,8 +1865,8 @@ convert_EXISTS_to_ANY(PlannerInfo *root, Query *subselect,
/*
* Replace correlation vars (uplevel vars) with Params.
*
- * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions, and
- * MergeSupportFuncs are replaced, too.
+ * Uplevel PlaceHolderVars, aggregates, GROUPING() expressions,
+ * MergeSupportFuncs, and ReturningExprs are replaced, too.
*
* Note: it is critical that this runs immediately after SS_process_sublinks.
* Since we do not recurse into the arguments of uplevel PHVs and aggregates,
@@ -1924,6 +1926,12 @@ replace_correlation_vars_mutator(Node *node, PlannerInfo *root)
return (Node *) replace_outer_merge_support(root,
(MergeSupportFunc *) node);
}
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return (Node *) replace_outer_returning(root,
+ (ReturningExpr *) node);
+ }
return expression_tree_mutator(node, replace_correlation_vars_mutator, root);
}
@@ -1977,11 +1985,11 @@ process_sublinks_mutator(Node *node, process_sublinks_context *context)
}
/*
- * Don't recurse into the arguments of an outer PHV, Aggref or
- * GroupingFunc here. Any SubLinks in the arguments have to be dealt with
- * at the outer query level; they'll be handled when build_subplan
- * collects the PHV, Aggref or GroupingFunc into the arguments to be
- * passed down to the current subplan.
+ * Don't recurse into the arguments of an outer PHV, Aggref, GroupingFunc
+ * or ReturningExpr here. Any SubLinks in the arguments have to be dealt
+ * with at the outer query level; they'll be handled when build_subplan
+ * collects the PHV, Aggref, GroupingFunc or ReturningExpr into the
+ * arguments to be passed down to the current subplan.
*/
if (IsA(node, PlaceHolderVar))
{
@@ -1998,6 +2006,11 @@ process_sublinks_mutator(Node *node, process_sublinks_context *context)
if (((GroupingFunc *) node)->agglevelsup > 0)
return node;
}
+ else if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup > 0)
+ return node;
+ }
/*
* We should never see a SubPlan expression in the input (since this is
@@ -2110,7 +2123,9 @@ SS_identify_outer_params(PlannerInfo *root)
outer_params = NULL;
for (proot = root->parent_root; proot != NULL; proot = proot->parent_root)
{
- /* Include ordinary Var/PHV/Aggref/GroupingFunc params */
+ /*
+ * Include ordinary Var/PHV/Aggref/GroupingFunc/ReturningExpr params.
+ */
foreach(l, proot->plan_params)
{
PlannerParamItem *pitem = (PlannerParamItem *) lfirst(l);
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
index 82775a3dd5..5d9225e990 100644
--- a/src/backend/optimizer/prep/prepjointree.c
+++ b/src/backend/optimizer/prep/prepjointree.c
@@ -2539,7 +2539,8 @@ pullup_replace_vars_callback(Var *var,
* expansion with varlevelsup = 0, and then adjust below if needed.
*/
expandRTE(rcon->target_rte,
- var->varno, 0 /* not varlevelsup */ , var->location,
+ var->varno, 0 /* not varlevelsup */ ,
+ var->varreturningtype, var->location,
(var->vartype != RECORDOID),
&colnames, &fields);
/* Expand the generated per-field Vars, but don't insert PHVs there */
diff --git a/src/backend/optimizer/util/appendinfo.c b/src/backend/optimizer/util/appendinfo.c
index cece3a5be7..5b3dc0d865 100644
--- a/src/backend/optimizer/util/appendinfo.c
+++ b/src/backend/optimizer/util/appendinfo.c
@@ -253,6 +253,13 @@ adjust_appendrel_attrs_mutator(Node *node,
* all non-Var outputs of such subqueries, and then we could look up
* the pre-existing PHV here. Or perhaps just wrap the translations
* that way to begin with?
+ *
+ * If var->varreturningtype is not VAR_RETURNING_DEFAULT, then that
+ * also needs to be copied to the translated Var. That too would fail
+ * if the translation wasn't a Var, but that should never happen since
+ * a non-default var->varreturningtype is only used for Vars referring
+ * to the result relation, which should never be a flattened UNION ALL
+ * subquery.
*/
for (cnt = 0; cnt < nappinfos; cnt++)
@@ -283,9 +290,17 @@ adjust_appendrel_attrs_mutator(Node *node,
elog(ERROR, "attribute %d of relation \"%s\" does not exist",
var->varattno, get_rel_name(appinfo->parent_reloid));
if (IsA(newnode, Var))
+ {
+ ((Var *) newnode)->varreturningtype = var->varreturningtype;
((Var *) newnode)->varnullingrels = var->varnullingrels;
- else if (var->varnullingrels != NULL)
- elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
+ else
+ {
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
+ if (var->varnullingrels != NULL)
+ elog(ERROR, "failed to apply nullingrels to a non-Var");
+ }
return newnode;
}
else if (var->varattno == 0)
@@ -339,6 +354,8 @@ adjust_appendrel_attrs_mutator(Node *node,
rowexpr->colnames = copyObject(rte->eref->colnames);
rowexpr->location = -1;
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ elog(ERROR, "failed to apply returningtype to a non-Var");
if (var->varnullingrels != NULL)
elog(ERROR, "failed to apply nullingrels to a non-Var");
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index de1f340cbe..43dfecfb47 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -1295,6 +1295,7 @@ contain_leaked_vars_walker(Node *node, void *context)
case T_NullTest:
case T_BooleanTest:
case T_NextValueExpr:
+ case T_ReturningExpr:
case T_List:
/*
@@ -3404,6 +3405,8 @@ eval_const_expressions_mutator(Node *node,
fselect->resulttypmod,
fselect->resultcollid,
((Var *) arg)->varlevelsup);
+ /* New Var has same OLD/NEW returning as old one */
+ newvar->varreturningtype = ((Var *) arg)->varreturningtype;
/* New Var is nullable by same rels as the old one */
newvar->varnullingrels = ((Var *) arg)->varnullingrels;
return (Node *) newvar;
diff --git a/src/backend/optimizer/util/paramassign.c b/src/backend/optimizer/util/paramassign.c
index 8e089c2707..3bd3ce37c8 100644
--- a/src/backend/optimizer/util/paramassign.c
+++ b/src/backend/optimizer/util/paramassign.c
@@ -91,6 +91,7 @@ assign_param_for_var(PlannerInfo *root, Var *var)
pvar->vartype == var->vartype &&
pvar->vartypmod == var->vartypmod &&
pvar->varcollid == var->varcollid &&
+ pvar->varreturningtype == var->varreturningtype &&
bms_equal(pvar->varnullingrels, var->varnullingrels))
return pitem->paramId;
}
@@ -358,6 +359,52 @@ replace_outer_merge_support(PlannerInfo *root, MergeSupportFunc *msf)
return retval;
}
+/*
+ * Generate a Param node to replace the given ReturningExpr expression which
+ * is expected to have retlevelsup > 0 (ie, it is not local). Record the need
+ * for the ReturningExpr in the proper upper-level root->plan_params.
+ */
+Param *
+replace_outer_returning(PlannerInfo *root, ReturningExpr *rexpr)
+{
+ Param *retval;
+ PlannerParamItem *pitem;
+ Index levelsup;
+ Oid ptype = exprType((Node *) rexpr->retexpr);
+
+ Assert(rexpr->retlevelsup > 0 && rexpr->retlevelsup < root->query_level);
+
+ /* Find the query level the ReturningExpr belongs to */
+ for (levelsup = rexpr->retlevelsup; levelsup > 0; levelsup--)
+ root = root->parent_root;
+
+ /*
+ * It does not seem worthwhile to try to de-duplicate references to outer
+ * ReturningExprs. Just make a new slot every time.
+ */
+ rexpr = copyObject(rexpr);
+ IncrementVarSublevelsUp((Node *) rexpr, -((int) rexpr->retlevelsup), 0);
+ Assert(rexpr->retlevelsup == 0);
+
+ pitem = makeNode(PlannerParamItem);
+ pitem->item = (Node *) rexpr;
+ pitem->paramId = list_length(root->glob->paramExecTypes);
+ root->glob->paramExecTypes = lappend_oid(root->glob->paramExecTypes,
+ ptype);
+
+ root->plan_params = lappend(root->plan_params, pitem);
+
+ retval = makeNode(Param);
+ retval->paramkind = PARAM_EXEC;
+ retval->paramid = pitem->paramId;
+ retval->paramtype = ptype;
+ retval->paramtypmod = exprTypmod((Node *) rexpr->retexpr);
+ retval->paramcollid = exprCollation((Node *) rexpr->retexpr);
+ retval->location = exprLocation((Node *) rexpr->retexpr);
+
+ return retval;
+}
+
/*
* Generate a Param node to replace the given Var,
* which is expected to come from some upper NestLoop plan node.
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index b9759c3125..ab63321659 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1846,8 +1846,8 @@ build_physical_tlist(PlannerInfo *root, RelOptInfo *rel)
case RTE_NAMEDTUPLESTORE:
case RTE_RESULT:
/* Not all of these can have dropped cols, but share code anyway */
- expandRTE(rte, varno, 0, -1, true /* include dropped */ ,
- NULL, &colvars);
+ expandRTE(rte, varno, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , NULL, &colvars);
foreach(l, colvars)
{
var = (Var *) lfirst(l);
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
index 367d080ccf..8065237a18 100644
--- a/src/backend/optimizer/util/var.c
+++ b/src/backend/optimizer/util/var.c
@@ -76,6 +76,7 @@ static bool pull_varattnos_walker(Node *node, pull_varattnos_context *context);
static bool pull_vars_walker(Node *node, pull_vars_context *context);
static bool contain_var_clause_walker(Node *node, void *context);
static bool contain_vars_of_level_walker(Node *node, int *sublevels_up);
+static bool contain_vars_returning_old_or_new_walker(Node *node, void *context);
static bool locate_var_of_level_walker(Node *node,
locate_var_of_level_context *context);
static bool pull_var_clause_walker(Node *node,
@@ -492,6 +493,49 @@ contain_vars_of_level_walker(Node *node, int *sublevels_up)
}
+/*
+ * contain_vars_returning_old_or_new
+ * Recursively scan a clause to discover whether it contains any Var nodes
+ * (of the current query level) whose varreturningtype is VAR_RETURNING_OLD
+ * or VAR_RETURNING_NEW.
+ *
+ * Returns true if any found.
+ *
+ * Any ReturningExprs are also detected --- if an OLD/NEW Var was rewritten,
+ * we still regard this as a clause that returns OLD/NEW values.
+ *
+ * Does not examine subqueries, therefore must only be used after reduction
+ * of sublinks to subplans!
+ */
+bool
+contain_vars_returning_old_or_new(Node *node)
+{
+ return contain_vars_returning_old_or_new_walker(node, NULL);
+}
+
+static bool
+contain_vars_returning_old_or_new_walker(Node *node, void *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ if (((Var *) node)->varlevelsup == 0 &&
+ ((Var *) node)->varreturningtype != VAR_RETURNING_DEFAULT)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ if (IsA(node, ReturningExpr))
+ {
+ if (((ReturningExpr *) node)->retlevelsup == 0)
+ return true; /* abort the tree traversal and return true */
+ return false;
+ }
+ return expression_tree_walker(node, contain_vars_returning_old_or_new_walker,
+ context);
+}
+
+
/*
* locate_var_of_level
* Find the parse location of any Var of the specified query level.
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index 561cf4d6a7..0a7d14344d 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -641,8 +641,8 @@ transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -1054,7 +1054,7 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
* contain only the target relation, removing any entries added in a
* sub-SELECT or VALUES list.
*/
- if (stmt->onConflictClause || stmt->returningList)
+ if (stmt->onConflictClause || stmt->returningClause)
{
pstate->p_namespace = NIL;
addNSItemToQuery(pstate, pstate->p_target_nsitem,
@@ -1067,10 +1067,9 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
stmt->onConflictClause);
/* Process RETURNING, if any. */
- if (stmt->returningList)
- qry->returningList = transformReturningList(pstate,
- stmt->returningList,
- EXPR_KIND_RETURNING);
+ if (stmt->returningClause)
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/* done building the range table and jointree */
qry->rtable = pstate->p_rtable;
@@ -2548,8 +2547,8 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
qual = transformWhereClause(pstate, stmt->whereClause,
EXPR_KIND_WHERE, "WHERE");
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_RETURNING);
/*
* Now we are done with SELECT-like processing, and can get on with
@@ -2645,18 +2644,120 @@ transformUpdateTargetList(ParseState *pstate, List *origTlist)
}
/*
- * transformReturningList -
+ * addNSItemForReturning -
+ * add a ParseNamespaceItem for the OLD or NEW alias in RETURNING.
+ */
+static void
+addNSItemForReturning(ParseState *pstate, const char *aliasname,
+ VarReturningType returning_type)
+{
+ List *colnames;
+ int numattrs;
+ ParseNamespaceColumn *nscolumns;
+ ParseNamespaceItem *nsitem;
+
+ /* copy per-column data from the target relation */
+ colnames = pstate->p_target_nsitem->p_rte->eref->colnames;
+ numattrs = list_length(colnames);
+
+ nscolumns = (ParseNamespaceColumn *)
+ palloc(numattrs * sizeof(ParseNamespaceColumn));
+
+ memcpy(nscolumns, pstate->p_target_nsitem->p_nscolumns,
+ numattrs * sizeof(ParseNamespaceColumn));
+
+ /* mark all columns as returning OLD/NEW */
+ for (int i = 0; i < numattrs; i++)
+ nscolumns[i].p_varreturningtype = returning_type;
+
+ /* build the nsitem, copying most fields from the target relation */
+ nsitem = (ParseNamespaceItem *) palloc(sizeof(ParseNamespaceItem));
+ nsitem->p_names = makeAlias(aliasname, colnames);
+ nsitem->p_rte = pstate->p_target_nsitem->p_rte;
+ nsitem->p_rtindex = pstate->p_target_nsitem->p_rtindex;
+ nsitem->p_perminfo = pstate->p_target_nsitem->p_perminfo;
+ nsitem->p_nscolumns = nscolumns;
+ nsitem->p_returning_type = returning_type;
+
+ /* add it to the query namespace as a table-only item */
+ addNSItemToQuery(pstate, nsitem, false, true, false);
+}
+
+/*
+ * transformReturningClause -
* handle a RETURNING clause in INSERT/UPDATE/DELETE/MERGE
*/
-List *
-transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind)
+void
+transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind)
{
- List *rlist;
+ int save_nslen;
int save_next_resno;
- if (returningList == NIL)
- return NIL; /* nothing to do */
+ if (returningClause == NULL)
+ return; /* nothing to do */
+
+ /*
+ * Scan RETURNING WITH(...) options for OLD/NEW alias names. Complain if
+ * there is any conflict with existing relations.
+ */
+ foreach_node(ReturningOption, option, returningClause->options)
+ {
+ switch (option->option)
+ {
+ case RETURNING_OPTION_OLD:
+ if (qry->returningOldAlias != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "OLD"),
+ parser_errposition(pstate, option->location));
+ qry->returningOldAlias = option->value;
+ break;
+
+ case RETURNING_OPTION_NEW:
+ if (qry->returningNewAlias != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is OLD or NEW */
+ errmsg("%s cannot be specified multiple times", "NEW"),
+ parser_errposition(pstate, option->location));
+ qry->returningNewAlias = option->value;
+ break;
+
+ default:
+ elog(ERROR, "unrecognized returning option: %d", option->option);
+ }
+
+ if (refnameNamespaceItem(pstate, NULL, option->value, -1, NULL) != NULL)
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_ALIAS),
+ errmsg("table name \"%s\" specified more than once",
+ option->value),
+ parser_errposition(pstate, option->location));
+ }
+
+ /*
+ * If no OLD/NEW aliases specified, use "old"/"new" unless masked by
+ * existing relations.
+ */
+ if (qry->returningOldAlias == NULL &&
+ refnameNamespaceItem(pstate, NULL, "old", -1, NULL) == NULL)
+ qry->returningOldAlias = "old";
+ if (qry->returningNewAlias == NULL &&
+ refnameNamespaceItem(pstate, NULL, "new", -1, NULL) == NULL)
+ qry->returningNewAlias = "new";
+
+ /*
+ * Add the OLD and NEW aliases to the query namespace, for use in
+ * expressions in the RETURNING list.
+ */
+ save_nslen = list_length(pstate->p_namespace);
+ if (qry->returningOldAlias != NULL)
+ addNSItemForReturning(pstate, qry->returningOldAlias, VAR_RETURNING_OLD);
+ if (qry->returningNewAlias != NULL)
+ addNSItemForReturning(pstate, qry->returningNewAlias, VAR_RETURNING_NEW);
/*
* We need to assign resnos starting at one in the RETURNING list. Save
@@ -2666,8 +2767,10 @@ transformReturningList(ParseState *pstate, List *returningList,
save_next_resno = pstate->p_next_resno;
pstate->p_next_resno = 1;
- /* transform RETURNING identically to a SELECT targetlist */
- rlist = transformTargetList(pstate, returningList, exprKind);
+ /* transform RETURNING expressions identically to a SELECT targetlist */
+ qry->returningList = transformTargetList(pstate,
+ returningClause->exprs,
+ exprKind);
/*
* Complain if the nonempty tlist expanded to nothing (which is possible
@@ -2675,24 +2778,23 @@ transformReturningList(ParseState *pstate, List *returningList,
* allow this, the parsed Query will look like it didn't have RETURNING,
* with results that would probably surprise the user.
*/
- if (rlist == NIL)
+ if (qry->returningList == NIL)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("RETURNING must have at least one column"),
parser_errposition(pstate,
- exprLocation(linitial(returningList)))));
+ exprLocation(linitial(returningClause->exprs)))));
/* mark column origins */
- markTargetListOrigins(pstate, rlist);
+ markTargetListOrigins(pstate, qry->returningList);
/* resolve any still-unresolved output columns as being type text */
if (pstate->p_resolve_unknowns)
- resolveTargetListUnknowns(pstate, rlist);
+ resolveTargetListUnknowns(pstate, qry->returningList);
/* restore state */
+ pstate->p_namespace = list_truncate(pstate->p_namespace, save_nslen);
pstate->p_next_resno = save_next_resno;
-
- return rlist;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b4c1e2c69d..fef6414c62 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -265,6 +265,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
MergeWhenClause *mergewhen;
struct KeyActions *keyactions;
struct KeyAction *keyaction;
+ ReturningClause *retclause;
+ ReturningOptionKind retoptionkind;
}
%type <node> stmt toplevel_stmt schema_stmt routine_body_stmt
@@ -434,7 +436,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
opclass_purpose opt_opfamily transaction_mode_list_or_empty
OptTableFuncElementList TableFuncElementList opt_type_modifiers
prep_type_clause
- execute_param_clause using_clause returning_clause
+ execute_param_clause using_clause
+ returning_with_clause returning_options
opt_enum_val_list enum_val_list table_func_column_list
create_generic_options alter_generic_options
relation_expr_list dostmt_opt_list
@@ -443,6 +446,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
vacuum_relation_list opt_vacuum_relation_list
drop_option_list pub_obj_list
+%type <retclause> returning_clause
+%type <node> returning_option
+%type <retoptionkind> returning_option_kind
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
@@ -12177,7 +12183,7 @@ InsertStmt:
{
$5->relation = $4;
$5->onConflictClause = $6;
- $5->returningList = $7;
+ $5->returningClause = $7;
$5->withClause = $1;
$5->stmt_location = @$;
$$ = (Node *) $5;
@@ -12311,8 +12317,45 @@ opt_conf_expr:
;
returning_clause:
- RETURNING target_list { $$ = $2; }
- | /* EMPTY */ { $$ = NIL; }
+ RETURNING returning_with_clause target_list
+ {
+ ReturningClause *n = makeNode(ReturningClause);
+
+ n->options = $2;
+ n->exprs = $3;
+ $$ = n;
+ }
+ | /* EMPTY */
+ {
+ $$ = NULL;
+ }
+ ;
+
+returning_with_clause:
+ WITH '(' returning_options ')' { $$ = $3; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+returning_options:
+ returning_option { $$ = list_make1($1); }
+ | returning_options ',' returning_option { $$ = lappend($1, $3); }
+ ;
+
+returning_option:
+ returning_option_kind AS ColId
+ {
+ ReturningOption *n = makeNode(ReturningOption);
+
+ n->option = $1;
+ n->value = $3;
+ n->location = @1;
+ $$ = (Node *) n;
+ }
+ ;
+
+returning_option_kind:
+ OLD { $$ = RETURNING_OPTION_OLD; }
+ | NEW { $$ = RETURNING_OPTION_NEW; }
;
@@ -12331,7 +12374,7 @@ DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
n->relation = $4;
n->usingClause = $5;
n->whereClause = $6;
- n->returningList = $7;
+ n->returningClause = $7;
n->withClause = $1;
n->stmt_location = @$;
$$ = (Node *) n;
@@ -12406,7 +12449,7 @@ UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
n->targetList = $5;
n->fromClause = $6;
n->whereClause = $7;
- n->returningList = $8;
+ n->returningClause = $8;
n->withClause = $1;
n->stmt_location = @$;
$$ = (Node *) n;
@@ -12485,7 +12528,7 @@ MergeStmt:
m->sourceRelation = $6;
m->joinCondition = $8;
m->mergeWhenClauses = $9;
- m->returningList = $10;
+ m->returningClause = $10;
m->stmt_location = @$;
$$ = (Node *) m;
diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c
index 75a1bbfd89..2e64fcae7b 100644
--- a/src/backend/parser/parse_clause.c
+++ b/src/backend/parser/parse_clause.c
@@ -1585,6 +1585,7 @@ transformFromClauseItem(ParseState *pstate, Node *n,
jnsitem->p_cols_visible = true;
jnsitem->p_lateral_only = false;
jnsitem->p_lateral_ok = true;
+ jnsitem->p_returning_type = VAR_RETURNING_DEFAULT;
/* Per SQL, we must check for alias conflicts */
checkNameSpaceConflicts(pstate, list_make1(jnsitem), my_namespace);
my_namespace = lappend(my_namespace, jnsitem);
@@ -1647,6 +1648,7 @@ buildVarFromNSColumn(ParseState *pstate, ParseNamespaceColumn *nscol)
nscol->p_varcollid,
0);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index d1f64f8f0a..6d0cc1b0b7 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -2619,6 +2619,13 @@ transformWholeRowRef(ParseState *pstate, ParseNamespaceItem *nsitem,
* point, there seems no harm in expanding it now rather than during
* planning.
*
+ * Note that if the nsitem is an OLD/NEW alias for the target RTE (as can
+ * appear in a RETURNING list), its alias won't match the target RTE's
+ * alias, but we still want to make a whole-row Var here rather than a
+ * RowExpr, for consistency with direct references to the target RTE, and
+ * so that any dropped columns are handled correctly. Thus we also check
+ * p_returning_type here.
+ *
* Note that if the RTE is a function returning scalar, we create just a
* plain reference to the function value, not a composite containing a
* single column. This is pretty inconsistent at first sight, but it's
@@ -2626,13 +2633,17 @@ transformWholeRowRef(ParseState *pstate, ParseNamespaceItem *nsitem,
* "rel.*" mean the same thing for composite relations, so why not for
* scalar functions...
*/
- if (nsitem->p_names == nsitem->p_rte->eref)
+ if (nsitem->p_names == nsitem->p_rte->eref ||
+ nsitem->p_returning_type != VAR_RETURNING_DEFAULT)
{
Var *result;
result = makeWholeRowVar(nsitem->p_rte, nsitem->p_rtindex,
sublevels_up, true);
+ /* mark Var for RETURNING OLD/NEW, as necessary */
+ result->varreturningtype = nsitem->p_returning_type;
+
/* location is not filled in by makeWholeRowVar */
result->location = location;
@@ -2655,9 +2666,8 @@ transformWholeRowRef(ParseState *pstate, ParseNamespaceItem *nsitem,
* are in the RTE. We needn't worry about marking the RTE for SELECT
* access, as the common columns are surely so marked already.
*/
- expandRTE(nsitem->p_rte, nsitem->p_rtindex,
- sublevels_up, location, false,
- NULL, &fields);
+ expandRTE(nsitem->p_rte, nsitem->p_rtindex, sublevels_up,
+ nsitem->p_returning_type, location, false, NULL, &fields);
rowexpr = makeNode(RowExpr);
rowexpr->args = list_truncate(fields,
list_length(nsitem->p_names->colnames));
diff --git a/src/backend/parser/parse_merge.c b/src/backend/parser/parse_merge.c
index f92bef99d5..51d7703eff 100644
--- a/src/backend/parser/parse_merge.c
+++ b/src/backend/parser/parse_merge.c
@@ -247,8 +247,8 @@ transformMergeStmt(ParseState *pstate, MergeStmt *stmt)
qry->jointree = makeFromExpr(pstate->p_joinlist, NULL);
/* Transform the RETURNING list, if any */
- qry->returningList = transformReturningList(pstate, stmt->returningList,
- EXPR_KIND_MERGE_RETURNING);
+ transformReturningClause(pstate, qry, stmt->returningClause,
+ EXPR_KIND_MERGE_RETURNING);
/*
* We now have a good query shape, so now look at the WHEN conditions and
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 92a04e35df..679bf640c6 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -91,11 +91,13 @@ static void markRTEForSelectPriv(ParseState *pstate,
int rtindex, AttrNumber col);
static void expandRelation(Oid relid, Alias *eref,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static void expandTupleDesc(TupleDesc tupdesc, Alias *eref,
int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
static int specialAttNum(const char *attname);
@@ -763,6 +765,9 @@ scanNSItemForColumn(ParseState *pstate, ParseNamespaceItem *nsitem,
}
var->location = location;
+ /* Mark Var for RETURNING OLD/NEW, as necessary */
+ var->varreturningtype = nsitem->p_returning_type;
+
/* Mark Var if it's nulled by any outer joins */
markNullableIfNeeded(pstate, var);
@@ -1336,6 +1341,7 @@ buildNSItemFromTupleDesc(RangeTblEntry *rte, Index rtindex,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -1399,6 +1405,7 @@ buildNSItemFromLists(RangeTblEntry *rte, Index rtindex,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2300,6 +2307,7 @@ addRangeTableEntryForJoin(ParseState *pstate,
nsitem->p_cols_visible = true;
nsitem->p_lateral_only = false;
nsitem->p_lateral_ok = true;
+ nsitem->p_returning_type = VAR_RETURNING_DEFAULT;
return nsitem;
}
@@ -2720,9 +2728,10 @@ addNSItemToQuery(ParseState *pstate, ParseNamespaceItem *nsitem,
* results. If include_dropped is true then empty strings and NULL constants
* (not Vars!) are returned for dropped columns.
*
- * rtindex, sublevels_up, and location are the varno, varlevelsup, and location
- * values to use in the created Vars. Ordinarily rtindex should match the
- * actual position of the RTE in its rangetable.
+ * rtindex, sublevels_up, returning_type, and location are the varno,
+ * varlevelsup, varreturningtype, and location values to use in the created
+ * Vars. Ordinarily rtindex should match the actual position of the RTE in
+ * its rangetable.
*
* The output lists go into *colnames and *colvars.
* If only one of the two kinds of output list is needed, pass NULL for the
@@ -2730,6 +2739,7 @@ addNSItemToQuery(ParseState *pstate, ParseNamespaceItem *nsitem,
*/
void
expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -2745,7 +2755,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
case RTE_RELATION:
/* Ordinary relation RTE */
expandRelation(rte->relid, rte->eref,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up, returning_type, location,
include_dropped, colnames, colvars);
break;
case RTE_SUBQUERY:
@@ -2792,6 +2802,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
exprTypmod((Node *) te->expr),
exprCollation((Node *) te->expr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2829,7 +2840,8 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
Assert(tupdesc);
expandTupleDesc(tupdesc, rte->eref,
rtfunc->funccolcount, atts_done,
- rtindex, sublevels_up, location,
+ rtindex, sublevels_up,
+ returning_type, location,
include_dropped, colnames, colvars);
}
else if (functypclass == TYPEFUNC_SCALAR)
@@ -2849,6 +2861,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
exprTypmod(rtfunc->funcexpr),
exprCollation(rtfunc->funcexpr),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -2891,6 +2904,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
attrtypmod,
attrcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
}
@@ -2920,6 +2934,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
InvalidOid,
sublevels_up);
+ varnode->varreturningtype = returning_type;
*colvars = lappend(*colvars, varnode);
}
}
@@ -3002,6 +3017,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
exprTypmod(avar),
exprCollation(avar),
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3057,6 +3073,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
varnode = makeVar(rtindex, varattno,
coltype, coltypmod, colcoll,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3089,6 +3106,7 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
*/
static void
expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3097,7 +3115,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
/* Get the tupledesc and turn it over to expandTupleDesc */
rel = relation_open(relid, AccessShareLock);
expandTupleDesc(rel->rd_att, eref, rel->rd_att->natts, 0,
- rtindex, sublevels_up,
+ rtindex, sublevels_up, returning_type,
location, include_dropped,
colnames, colvars);
relation_close(rel, AccessShareLock);
@@ -3115,6 +3133,7 @@ expandRelation(Oid relid, Alias *eref, int rtindex, int sublevels_up,
static void
expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars)
{
@@ -3175,6 +3194,7 @@ expandTupleDesc(TupleDesc tupdesc, Alias *eref, int count, int offset,
attr->atttypid, attr->atttypmod,
attr->attcollation,
sublevels_up);
+ varnode->varreturningtype = returning_type;
varnode->location = location;
*colvars = lappend(*colvars, varnode);
@@ -3227,6 +3247,7 @@ expandNSItemVars(ParseState *pstate, ParseNamespaceItem *nsitem,
nscol->p_varcollid,
sublevels_up);
/* makeVar doesn't offer parameters for these, so set by hand: */
+ var->varreturningtype = nscol->p_varreturningtype;
var->varnosyn = nscol->p_varnosyn;
var->varattnosyn = nscol->p_varattnosyn;
var->location = location;
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
index 93915031be..4aba0d9d4d 100644
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -1550,8 +1550,8 @@ expandRecordVariable(ParseState *pstate, Var *var, int levelsup)
*lvar;
int i;
- expandRTE(rte, var->varno, 0, var->location, false,
- &names, &vars);
+ expandRTE(rte, var->varno, 0, var->varreturningtype,
+ var->location, false, &names, &vars);
tupleDesc = CreateTemplateTupleDesc(list_length(vars));
i = 1;
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index 1a5dfd0aa4..b74f2acc32 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -641,6 +641,7 @@ rewriteRuleAction(Query *parsetree,
0,
rt_fetch(new_varno, sub_action->rtable),
parsetree->targetList,
+ sub_action->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -674,10 +675,15 @@ rewriteRuleAction(Query *parsetree,
rt_fetch(parsetree->resultRelation,
parsetree->rtable),
rule_action->returningList,
+ rule_action->resultRelation,
REPLACEVARS_REPORT_ERROR,
0,
&rule_action->hasSubLinks);
+ /* use triggering query's aliases for OLD and NEW in RETURNING list */
+ rule_action->returningOldAlias = parsetree->returningOldAlias;
+ rule_action->returningNewAlias = parsetree->returningNewAlias;
+
/*
* There could have been some SubLinks in parsetree's returningList,
* in which case we'd better mark the rule_action correctly.
@@ -2358,6 +2364,7 @@ CopyAndAddInvertedQual(Query *parsetree,
rt_fetch(rt_index,
parsetree->rtable),
parsetree->targetList,
+ parsetree->resultRelation,
(event == CMD_UPDATE) ?
REPLACEVARS_CHANGE_VARNO :
REPLACEVARS_SUBSTITUTE_NULL,
@@ -3582,6 +3589,7 @@ rewriteTargetView(Query *parsetree, Relation view)
0,
view_rte,
view_targetlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
NULL);
@@ -3733,6 +3741,7 @@ rewriteTargetView(Query *parsetree, Relation view)
0,
view_rte,
tmp_tlist,
+ new_rt_index,
REPLACEVARS_REPORT_ERROR,
0,
&parsetree->hasSubLinks);
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
index 047396e390..ef1a8d9de4 100644
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -810,6 +810,14 @@ IncrementVarSublevelsUp_walker(Node *node,
phv->phlevelsup += context->delta_sublevels_up;
/* fall through to recurse into argument */
}
+ if (IsA(node, ReturningExpr))
+ {
+ ReturningExpr *rexpr = (ReturningExpr *) node;
+
+ if (rexpr->retlevelsup >= context->min_sublevels_up)
+ rexpr->retlevelsup += context->delta_sublevels_up;
+ /* fall through to recurse into argument */
+ }
if (IsA(node, RangeTblEntry))
{
RangeTblEntry *rte = (RangeTblEntry *) node;
@@ -875,6 +883,67 @@ IncrementVarSublevelsUp_rtable(List *rtable, int delta_sublevels_up,
QTW_EXAMINE_RTES_BEFORE);
}
+/*
+ * SetVarReturningType - adjust Var nodes for a specified varreturningtype.
+ *
+ * Find all Var nodes referring to the specified result relation in the given
+ * expression and set their varreturningtype to the specified value.
+ *
+ * NOTE: although this has the form of a walker, we cheat and modify the
+ * Var nodes in-place. The given expression tree should have been copied
+ * earlier to ensure that no unwanted side-effects occur!
+ */
+
+typedef struct
+{
+ int result_relation;
+ int sublevels_up;
+ VarReturningType returning_type;
+} SetVarReturningType_context;
+
+static bool
+SetVarReturningType_walker(Node *node, SetVarReturningType_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, Var))
+ {
+ Var *var = (Var *) node;
+
+ if (var->varno == context->result_relation &&
+ var->varlevelsup == context->sublevels_up)
+ var->varreturningtype = context->returning_type;
+
+ return false;
+ }
+
+ if (IsA(node, Query))
+ {
+ /* Recurse into subselects */
+ bool result;
+
+ context->sublevels_up++;
+ result = query_tree_walker((Query *) node, SetVarReturningType_walker,
+ context, 0);
+ context->sublevels_up--;
+ return result;
+ }
+ return expression_tree_walker(node, SetVarReturningType_walker, context);
+}
+
+static void
+SetVarReturningType(Node *node, int result_relation, int sublevels_up,
+ VarReturningType returning_type)
+{
+ SetVarReturningType_context context;
+
+ context.result_relation = result_relation;
+ context.sublevels_up = sublevels_up;
+ context.returning_type = returning_type;
+
+ /* Expect to start with an expression */
+ SetVarReturningType_walker(node, &context);
+}
/*
* rangeTableEntry_used - detect whether an RTE is referenced somewhere
@@ -1640,6 +1709,15 @@ map_variable_attnos(Node *node,
* relation. This is needed to handle whole-row Vars referencing the target.
* We expand such Vars into RowExpr constructs.
*
+ * In addition, for INSERT/UPDATE/DELETE/MERGE queries, the caller must
+ * provide result_relation, the index of the result relation in the rewritten
+ * query. This is needed to handle OLD/NEW RETURNING list Vars referencing
+ * target_varno. When such Vars are expanded, their varreturningtype is
+ * copied onto any replacement Vars referencing result_relation. In addition,
+ * if the replacement expression from the targetlist is not simply a Var
+ * referencing result_relation, it is wrapped in a ReturningExpr node (causing
+ * the executor to return NULL if the OLD/NEW row doesn't exist).
+ *
* outer_hasSubLinks works the same as for replace_rte_variables().
*/
@@ -1647,6 +1725,7 @@ typedef struct
{
RangeTblEntry *target_rte;
List *targetlist;
+ int result_relation;
ReplaceVarsNoMatchOption nomatch_option;
int nomatch_varno;
} ReplaceVarsFromTargetList_context;
@@ -1671,10 +1750,13 @@ ReplaceVarsFromTargetList_callback(Var *var,
* dropped columns. If the var is RECORD (ie, this is a JOIN), then
* omit dropped columns. In the latter case, attach column names to
* the RowExpr for use of the executor and ruleutils.c.
+ *
+ * The varreturningtype is copied onto each individual field Var, so
+ * that it is handled correctly when we recurse.
*/
expandRTE(rcon->target_rte,
- var->varno, var->varlevelsup, var->location,
- (var->vartype != RECORDOID),
+ var->varno, var->varlevelsup, var->varreturningtype,
+ var->location, (var->vartype != RECORDOID),
&colnames, &fields);
/* Adjust the generated per-field Vars... */
fields = (List *) replace_rte_variables_mutator((Node *) fields,
@@ -1686,6 +1768,18 @@ ReplaceVarsFromTargetList_callback(Var *var,
rowexpr->colnames = (var->vartype == RECORDOID) ? colnames : NIL;
rowexpr->location = var->location;
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = var->varreturningtype == VAR_RETURNING_OLD;
+ rexpr->retexpr = (Expr *) rowexpr;
+
+ return (Node *) rexpr;
+ }
+
return (Node *) rowexpr;
}
@@ -1751,6 +1845,34 @@ ReplaceVarsFromTargetList_callback(Var *var,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("NEW variables in ON UPDATE rules cannot reference columns that are part of a multiple assignment in the subject UPDATE command")));
+ /* Handle any OLD/NEW RETURNING list Vars */
+ if (var->varreturningtype != VAR_RETURNING_DEFAULT)
+ {
+ /*
+ * Copy varreturningtype onto any Vars in the tlist item that
+ * refer to result_relation (which had better be non-zero).
+ */
+ if (rcon->result_relation == 0)
+ elog(ERROR, "variable returning old/new found outside RETURNING list");
+
+ SetVarReturningType((Node *) newnode, rcon->result_relation,
+ var->varlevelsup, var->varreturningtype);
+
+ /* Wrap it in a ReturningExpr, if needed, per comments above */
+ if (!IsA(newnode, Var) ||
+ ((Var *) newnode)->varno != rcon->result_relation ||
+ ((Var *) newnode)->varlevelsup != var->varlevelsup)
+ {
+ ReturningExpr *rexpr = makeNode(ReturningExpr);
+
+ rexpr->retlevelsup = var->varlevelsup;
+ rexpr->retold = (var->varreturningtype == VAR_RETURNING_OLD);
+ rexpr->retexpr = newnode;
+
+ newnode = (Expr *) rexpr;
+ }
+ }
+
return (Node *) newnode;
}
}
@@ -1760,6 +1882,7 @@ ReplaceVarsFromTargetList(Node *node,
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks)
@@ -1768,6 +1891,7 @@ ReplaceVarsFromTargetList(Node *node,
context.target_rte = target_rte;
context.targetlist = targetlist;
+ context.result_relation = result_relation;
context.nomatch_option = nomatch_option;
context.nomatch_varno = nomatch_varno;
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 2089b52d57..e2df4b1e19 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -167,6 +167,8 @@ typedef struct
List *subplans; /* List of Plan trees for SubPlans */
List *ctes; /* List of CommonTableExpr nodes */
AppendRelInfo **appendrels; /* Array of AppendRelInfo nodes, or NULL */
+ char *ret_old_alias; /* alias for OLD in RETURNING list */
+ char *ret_new_alias; /* alias for NEW in RETURNING list */
/* Workspace for column alias assignment: */
bool unique_using; /* Are we making USING names globally unique */
List *using_names; /* List of assigned names for USING columns */
@@ -426,6 +428,7 @@ static void get_merge_query_def(Query *query, deparse_context *context);
static void get_utility_query_def(Query *query, deparse_context *context);
static void get_basic_select_query(Query *query, deparse_context *context);
static void get_target_list(List *targetList, deparse_context *context);
+static void get_returning_clause(Query *query, deparse_context *context);
static void get_setop_query(Node *setOp, Query *query,
deparse_context *context);
static Node *get_rule_sortgroupclause(Index ref, List *tlist,
@@ -3800,6 +3803,10 @@ deparse_context_for_plan_tree(PlannedStmt *pstmt, List *rtable_names)
* the most-closely-nested first. This is needed to resolve PARAM_EXEC
* Params. Note we assume that all the Plan nodes share the same rtable.
*
+ * For a ModifyTable plan, we might also need to resolve references to OLD/NEW
+ * variables in the RETURNING list, so we copy the alias names of the OLD and
+ * NEW rows from the ModifyTable plan node.
+ *
* Once this function has been called, deparse_expression() can be called on
* subsidiary expression(s) of the specified Plan node. To deparse
* expressions of a different Plan node in the same Plan tree, re-call this
@@ -3820,6 +3827,13 @@ set_deparse_context_plan(List *dpcontext, Plan *plan, List *ancestors)
dpns->ancestors = ancestors;
set_deparse_plan(dpns, plan);
+ /* For ModifyTable, set aliases for OLD and NEW in RETURNING */
+ if (IsA(plan, ModifyTable))
+ {
+ dpns->ret_old_alias = ((ModifyTable *) plan)->returningOldAlias;
+ dpns->ret_new_alias = ((ModifyTable *) plan)->returningNewAlias;
+ }
+
return dpcontext;
}
@@ -4017,6 +4031,8 @@ set_deparse_for_query(deparse_namespace *dpns, Query *query,
dpns->subplans = NIL;
dpns->ctes = query->cteList;
dpns->appendrels = NULL;
+ dpns->ret_old_alias = query->returningOldAlias;
+ dpns->ret_new_alias = query->returningNewAlias;
/* Assign a unique relation alias to each RTE */
set_rtable_names(dpns, parent_namespaces, NULL);
@@ -4411,8 +4427,8 @@ set_relation_column_names(deparse_namespace *dpns, RangeTblEntry *rte,
if (rte->rtekind == RTE_FUNCTION && rte->functions != NIL)
{
/* Since we're not creating Vars, rtindex etc. don't matter */
- expandRTE(rte, 1, 0, -1, true /* include dropped */ ,
- &colnames, NULL);
+ expandRTE(rte, 1, 0, VAR_RETURNING_DEFAULT, -1,
+ true /* include dropped */ , &colnames, NULL);
}
else
colnames = rte->eref->colnames;
@@ -6338,6 +6354,45 @@ get_target_list(List *targetList, deparse_context *context)
pfree(targetbuf.data);
}
+static void
+get_returning_clause(Query *query, deparse_context *context)
+{
+ StringInfo buf = context->buf;
+
+ if (query->returningList)
+ {
+ bool have_with = false;
+
+ appendContextKeyword(context, " RETURNING",
+ -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
+
+ /* Add WITH (OLD/NEW) options, if they're not the defaults */
+ if (query->returningOldAlias && strcmp(query->returningOldAlias, "old") != 0)
+ {
+ appendStringInfo(buf, " WITH (OLD AS %s",
+ quote_identifier(query->returningOldAlias));
+ have_with = true;
+ }
+ if (query->returningNewAlias && strcmp(query->returningNewAlias, "new") != 0)
+ {
+ if (have_with)
+ appendStringInfo(buf, ", NEW AS %s",
+ quote_identifier(query->returningNewAlias));
+ else
+ {
+ appendStringInfo(buf, " WITH (NEW AS %s",
+ quote_identifier(query->returningNewAlias));
+ have_with = true;
+ }
+ }
+ if (have_with)
+ appendStringInfoChar(buf, ')');
+
+ /* Add the returning expressions themselves */
+ get_target_list(query->returningList, context);
+ }
+}
+
static void
get_setop_query(Node *setOp, Query *query, deparse_context *context)
{
@@ -7018,11 +7073,7 @@ get_insert_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7074,11 +7125,7 @@ get_update_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7277,11 +7324,7 @@ get_delete_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7440,11 +7483,7 @@ get_merge_query_def(Query *query, deparse_context *context)
/* Add RETURNING if present */
if (query->returningList)
- {
- appendContextKeyword(context, " RETURNING",
- -PRETTYINDENT_STD, PRETTYINDENT_STD, 1);
- get_target_list(query->returningList, context);
- }
+ get_returning_clause(query, context);
}
@@ -7592,7 +7631,15 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
}
rte = rt_fetch(varno, dpns->rtable);
- refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+
+ /* might be returning old/new column value */
+ if (var->varreturningtype == VAR_RETURNING_OLD)
+ refname = dpns->ret_old_alias;
+ else if (var->varreturningtype == VAR_RETURNING_NEW)
+ refname = dpns->ret_new_alias;
+ else
+ refname = (char *) list_nth(dpns->rtable_names, varno - 1);
+
colinfo = deparse_columns_fetch(varno, dpns);
attnum = varattno;
}
@@ -7706,7 +7753,8 @@ get_variable(Var *var, int levelsup, bool istoplevel, deparse_context *context)
attname = get_rte_attribute_name(rte, attnum);
}
- need_prefix = (context->varprefix || attname == NULL);
+ need_prefix = (context->varprefix || attname == NULL ||
+ var->varreturningtype != VAR_RETURNING_DEFAULT);
/*
* If we're considering a plain Var in an ORDER BY (but not GROUP BY)
@@ -8803,6 +8851,9 @@ isSimpleNode(Node *node, Node *parentNode, int prettyFlags)
case T_ConvertRowtypeExpr:
return isSimpleNode((Node *) ((ConvertRowtypeExpr *) node)->arg,
node, prettyFlags);
+ case T_ReturningExpr:
+ return isSimpleNode((Node *) ((ReturningExpr *) node)->retexpr,
+ node, prettyFlags);
case T_OpExpr:
{
@@ -10288,6 +10339,20 @@ get_rule_expr(Node *node, deparse_context *context,
}
break;
+ case T_ReturningExpr:
+ {
+ ReturningExpr *retExpr = (ReturningExpr *) node;
+
+ /*
+ * We cannot see a ReturningExpr in rule deparsing, only while
+ * EXPLAINing a query plan (ReturningExpr nodes are only ever
+ * adding during query rewriting). Just display the expression
+ * returned (an expanded view column).
+ */
+ get_rule_expr((Node *) retExpr->retexpr, context, showimplicit);
+ }
+ break;
+
case T_PartitionBoundSpec:
{
PartitionBoundSpec *spec = (PartitionBoundSpec *) node;
diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h
index 8019e5490e..2841c69edd 100644
--- a/src/include/executor/execExpr.h
+++ b/src/include/executor/execExpr.h
@@ -26,9 +26,9 @@ struct JsonConstructorExprState;
/* Bits in ExprState->flags (see also execnodes.h for public flag bits): */
/* expression's interpreter has been initialized */
-#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 1)
+#define EEO_FLAG_INTERPRETER_INITIALIZED (1 << 5)
/* jump-threading is in use */
-#define EEO_FLAG_DIRECT_THREADED (1 << 2)
+#define EEO_FLAG_DIRECT_THREADED (1 << 6)
/* Typical API for out-of-line evaluation subroutines */
typedef void (*ExecEvalSubroutine) (ExprState *state,
@@ -72,16 +72,22 @@ typedef enum ExprEvalOp
EEOP_INNER_FETCHSOME,
EEOP_OUTER_FETCHSOME,
EEOP_SCAN_FETCHSOME,
+ EEOP_OLD_FETCHSOME,
+ EEOP_NEW_FETCHSOME,
/* compute non-system Var value */
EEOP_INNER_VAR,
EEOP_OUTER_VAR,
EEOP_SCAN_VAR,
+ EEOP_OLD_VAR,
+ EEOP_NEW_VAR,
/* compute system Var value */
EEOP_INNER_SYSVAR,
EEOP_OUTER_SYSVAR,
EEOP_SCAN_SYSVAR,
+ EEOP_OLD_SYSVAR,
+ EEOP_NEW_SYSVAR,
/* compute wholerow Var */
EEOP_WHOLEROW,
@@ -94,6 +100,8 @@ typedef enum ExprEvalOp
EEOP_ASSIGN_INNER_VAR,
EEOP_ASSIGN_OUTER_VAR,
EEOP_ASSIGN_SCAN_VAR,
+ EEOP_ASSIGN_OLD_VAR,
+ EEOP_ASSIGN_NEW_VAR,
/* assign ExprState's resvalue/resnull to a column of its resultslot */
EEOP_ASSIGN_TMP,
@@ -178,6 +186,7 @@ typedef enum ExprEvalOp
EEOP_SQLVALUEFUNCTION,
EEOP_CURRENTOFEXPR,
EEOP_NEXTVALUEEXPR,
+ EEOP_RETURNINGEXPR,
EEOP_ARRAYEXPR,
EEOP_ARRAYCOERCE,
EEOP_ROW,
@@ -301,7 +310,7 @@ typedef struct ExprEvalStep
*/
union
{
- /* for EEOP_INNER/OUTER/SCAN_FETCHSOME */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_FETCHSOME */
struct
{
/* attribute number up to which to fetch (inclusive) */
@@ -314,13 +323,14 @@ typedef struct ExprEvalStep
const TupleTableSlotOps *kind;
} fetch;
- /* for EEOP_INNER/OUTER/SCAN_[SYS]VAR[_FIRST] */
+ /* for EEOP_INNER/OUTER/SCAN/OLD/NEW_[SYS]VAR */
struct
{
/* attnum is attr number - 1 for regular VAR ... */
/* but it's just the normal (negative) attr number for SYSVAR */
int attnum;
Oid vartype; /* type OID of variable */
+ VarReturningType varreturningtype; /* return old/new/default */
} var;
/* for EEOP_WHOLEROW */
@@ -349,6 +359,13 @@ typedef struct ExprEvalStep
int resultnum;
} assign_tmp;
+ /* for EEOP_RETURNINGEXPR */
+ struct
+ {
+ uint8 nullflag; /* flag to test if OLD/NEW row is NULL */
+ int jumpdone; /* jump here if OLD/NEW row is NULL */
+ } returningexpr;
+
/* for EEOP_CONST */
struct
{
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f8a8d03e53..c7db6defd3 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -629,6 +629,7 @@ extern int ExecCleanTargetListLength(List *targetlist);
extern TupleTableSlot *ExecGetTriggerOldSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetTriggerNewSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleTableSlot *ExecGetReturningSlot(EState *estate, ResultRelInfo *relInfo);
+extern TupleTableSlot *ExecGetAllNullSlot(EState *estate, ResultRelInfo *relInfo);
extern TupleConversionMap *ExecGetChildToRootMap(ResultRelInfo *resultRelInfo);
extern TupleConversionMap *ExecGetRootToChildMap(ResultRelInfo *resultRelInfo, EState *estate);
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b3f7aa299f..d0f2dca592 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -74,11 +74,20 @@ typedef Datum (*ExprStateEvalFunc) (struct ExprState *expression,
/* Bits in ExprState->flags (see also execExpr.h for private flag bits): */
/* expression is for use with ExecQual() */
#define EEO_FLAG_IS_QUAL (1 << 0)
+/* expression refers to OLD table columns */
+#define EEO_FLAG_HAS_OLD (1 << 1)
+/* expression refers to NEW table columns */
+#define EEO_FLAG_HAS_NEW (1 << 2)
+/* OLD table row is NULL in RETURNING list */
+#define EEO_FLAG_OLD_IS_NULL (1 << 3)
+/* NEW table row is NULL in RETURNING list */
+#define EEO_FLAG_NEW_IS_NULL (1 << 4)
typedef struct ExprState
{
NodeTag type;
+#define FIELDNO_EXPRSTATE_FLAGS 1
uint8 flags; /* bitmask of EEO_FLAG_* bits, see above */
/*
@@ -290,6 +299,12 @@ typedef struct ExprContext
#define FIELDNO_EXPRCONTEXT_DOMAINNULL 13
bool domainValue_isNull;
+ /* Tuples that OLD/NEW Var nodes in RETURNING may refer to */
+#define FIELDNO_EXPRCONTEXT_OLDTUPLE 14
+ TupleTableSlot *ecxt_oldtuple;
+#define FIELDNO_EXPRCONTEXT_NEWTUPLE 15
+ TupleTableSlot *ecxt_newtuple;
+
/* Link to containing EState (NULL if a standalone ExprContext) */
struct EState *ecxt_estate;
@@ -504,6 +519,7 @@ typedef struct ResultRelInfo
TupleTableSlot *ri_ReturningSlot; /* for trigger output tuples */
TupleTableSlot *ri_TrigOldSlot; /* for a trigger's old tuple */
TupleTableSlot *ri_TrigNewSlot; /* for a trigger's new tuple */
+ TupleTableSlot *ri_AllNullSlot; /* for RETURNING OLD/NEW */
/* FDW callback functions, if foreign table */
struct FdwRoutine *ri_FdwRoutine;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 38d6ad7dcb..40a3d60063 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -197,6 +197,15 @@ typedef struct Query
OnConflictExpr *onConflict; /* ON CONFLICT DO [NOTHING | UPDATE] */
+ /*
+ * The following three fields describe the contents of the RETURNING list
+ * for INSERT/UPDATE/DELETE/MERGE. returningOldAlias and returningNewAlias
+ * are the alias names for OLD and NEW, which may be user-supplied values,
+ * the defaults "old" and "new", or NULL (if the default "old"/"new" is
+ * already in use as the alias for some other relation).
+ */
+ char *returningOldAlias pg_node_attr(query_jumble_ignore);
+ char *returningNewAlias pg_node_attr(query_jumble_ignore);
List *returningList; /* return-values list (of TargetEntry) */
List *groupClause; /* a list of SortGroupClause's */
@@ -1726,6 +1735,41 @@ typedef struct MergeWhenClause
List *values; /* VALUES to INSERT, or NULL */
} MergeWhenClause;
+/*
+ * ReturningOptionKind -
+ * Possible kinds of option in RETURNING WITH(...) list
+ *
+ * Currently, this is used only for specifying OLD/NEW aliases.
+ */
+typedef enum ReturningOptionKind
+{
+ RETURNING_OPTION_OLD, /* specify alias for OLD in RETURNING */
+ RETURNING_OPTION_NEW, /* specify alias for NEW in RETURNING */
+} ReturningOptionKind;
+
+/*
+ * ReturningOption -
+ * An individual option in the RETURNING WITH(...) list
+ */
+typedef struct ReturningOption
+{
+ NodeTag type;
+ ReturningOptionKind option; /* specified option */
+ char *value; /* option's value */
+ ParseLoc location; /* token location, or -1 if unknown */
+} ReturningOption;
+
+/*
+ * ReturningClause -
+ * List of RETURNING expressions, together with any WITH(...) options
+ */
+typedef struct ReturningClause
+{
+ NodeTag type;
+ List *options; /* list of ReturningOption elements */
+ List *exprs; /* list of expressions to return */
+} ReturningClause;
+
/*
* TriggerTransition -
* representation of transition row or table naming clause
@@ -2043,7 +2087,7 @@ typedef struct InsertStmt
List *cols; /* optional: names of the target columns */
Node *selectStmt; /* the source SELECT/VALUES, or NULL */
OnConflictClause *onConflictClause; /* ON CONFLICT clause */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
OverridingKind override; /* OVERRIDING clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
@@ -2060,7 +2104,7 @@ typedef struct DeleteStmt
RangeVar *relation; /* relation to delete from */
List *usingClause; /* optional using clause for more tables */
Node *whereClause; /* qualifications */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
@@ -2077,7 +2121,7 @@ typedef struct UpdateStmt
List *targetList; /* the target list (of ResTarget) */
Node *whereClause; /* qualifications */
List *fromClause; /* optional from clause for more tables */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
@@ -2094,7 +2138,7 @@ typedef struct MergeStmt
Node *sourceRelation; /* source relation */
Node *joinCondition; /* join condition between source and target */
List *mergeWhenClauses; /* list of MergeWhenClause(es) */
- List *returningList; /* list of expressions to return */
+ ReturningClause *returningClause; /* RETURNING clause */
WithClause *withClause; /* WITH clause */
ParseLoc stmt_location; /* start location, or -1 if unknown */
ParseLoc stmt_len; /* length in bytes; 0 means "rest of string" */
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index ef9ea7ee98..9e19cdd284 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -238,6 +238,8 @@ typedef struct ModifyTable
List *resultRelations; /* integer list of RT indexes */
List *updateColnosLists; /* per-target-table update_colnos lists */
List *withCheckOptionLists; /* per-target-table WCO lists */
+ char *returningOldAlias; /* alias for OLD in RETURNING lists */
+ char *returningNewAlias; /* alias for NEW in RETURNING lists */
List *returningLists; /* per-target-table RETURNING tlists */
List *fdwPrivLists; /* per-target-table FDW private data lists */
Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 9c2957eb54..d9a40ec89c 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -223,6 +223,11 @@ typedef struct Expr
* Note that it affects the meaning of all of varno, varnullingrels, and
* varnosyn, all of which refer to the range table of that query level.
*
+ * varreturningtype is used for Vars that refer to the target relation in the
+ * RETURNING list of data-modifying queries. The default behavior is to
+ * return old values for DELETE and new values for INSERT and UPDATE, but it
+ * is also possible to explicitly request old or new values.
+ *
* In the parser, varnosyn and varattnosyn are either identical to
* varno/varattno, or they specify the column's position in an aliased JOIN
* RTE that hides the semantic referent RTE's refname. This is a syntactic
@@ -244,6 +249,14 @@ typedef struct Expr
#define PRS2_OLD_VARNO 1
#define PRS2_NEW_VARNO 2
+/* Returning behavior for Vars in RETURNING list */
+typedef enum VarReturningType
+{
+ VAR_RETURNING_DEFAULT, /* return OLD for DELETE, else return NEW */
+ VAR_RETURNING_OLD, /* return OLD for DELETE/UPDATE, else NULL */
+ VAR_RETURNING_NEW, /* return NEW for INSERT/UPDATE, else NULL */
+} VarReturningType;
+
typedef struct Var
{
Expr xpr;
@@ -279,6 +292,9 @@ typedef struct Var
*/
Index varlevelsup;
+ /* returning type of this var (see above) */
+ VarReturningType varreturningtype;
+
/*
* varnosyn/varattnosyn are ignored for equality, because Vars with
* different syntactic identifiers are semantically the same as long as
@@ -2128,6 +2144,30 @@ typedef struct InferenceElem
Oid inferopclass; /* OID of att opclass, or InvalidOid */
} InferenceElem;
+/*
+ * ReturningExpr - return OLD/NEW.(expression) in RETURNING list
+ *
+ * This is used when updating an auto-updatable view and returning a view
+ * column that is not simply a Var referring to the base relation. In such
+ * cases, OLD/NEW.viewcol can expand to an arbitrary expression, but the
+ * result is required to be NULL if the OLD/NEW row doesn't exist. To handle
+ * this, the rewriter wraps the expanded expression in a ReturningExpr, which
+ * is equivalent to "CASE WHEN (OLD/NEW row exists) THEN (expr) ELSE NULL".
+ *
+ * A similar situation can arise when rewriting the RETURNING clause of a
+ * rule, which may also contain arbitrary expressions.
+ *
+ * ReturningExpr nodes never appear in a parsed Query --- they are only ever
+ * inserted by the rewriter.
+ */
+typedef struct ReturningExpr
+{
+ Expr xpr;
+ int retlevelsup; /* > 0 if it belongs to outer query */
+ bool retold; /* true for OLD, false for NEW */
+ Expr *retexpr; /* expression to be returned */
+} ReturningExpr;
+
/*--------------------
* TargetEntry -
* a target entry (used in query target lists)
diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h
index 734c82a27d..bcf8ed645c 100644
--- a/src/include/optimizer/optimizer.h
+++ b/src/include/optimizer/optimizer.h
@@ -199,6 +199,7 @@ extern void pull_varattnos(Node *node, Index varno, Bitmapset **varattnos);
extern List *pull_vars_of_level(Node *node, int levelsup);
extern bool contain_var_clause(Node *node);
extern bool contain_vars_of_level(Node *node, int levelsup);
+extern bool contain_vars_returning_old_or_new(Node *node);
extern int locate_var_of_level(Node *node, int levelsup);
extern List *pull_var_clause(Node *node, int flags);
extern Node *flatten_join_alias_vars(PlannerInfo *root, Query *query, Node *node);
diff --git a/src/include/optimizer/paramassign.h b/src/include/optimizer/paramassign.h
index 15321ebabb..59dcb1ff05 100644
--- a/src/include/optimizer/paramassign.h
+++ b/src/include/optimizer/paramassign.h
@@ -22,6 +22,8 @@ extern Param *replace_outer_agg(PlannerInfo *root, Aggref *agg);
extern Param *replace_outer_grouping(PlannerInfo *root, GroupingFunc *grp);
extern Param *replace_outer_merge_support(PlannerInfo *root,
MergeSupportFunc *msf);
+extern Param *replace_outer_returning(PlannerInfo *root,
+ ReturningExpr *rexpr);
extern Param *replace_nestloop_param_var(PlannerInfo *root, Var *var);
extern Param *replace_nestloop_param_placeholdervar(PlannerInfo *root,
PlaceHolderVar *phv);
diff --git a/src/include/parser/analyze.h b/src/include/parser/analyze.h
index ac49091264..f1bd18c49f 100644
--- a/src/include/parser/analyze.h
+++ b/src/include/parser/analyze.h
@@ -44,8 +44,9 @@ extern List *transformInsertRow(ParseState *pstate, List *exprlist,
bool strip_indirection);
extern List *transformUpdateTargetList(ParseState *pstate,
List *origTlist);
-extern List *transformReturningList(ParseState *pstate, List *returningList,
- ParseExprKind exprKind);
+extern void transformReturningClause(ParseState *pstate, Query *qry,
+ ReturningClause *returningClause,
+ ParseExprKind exprKind);
extern Query *transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree);
extern Query *transformStmt(ParseState *pstate, Node *parseTree);
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index 0de44d166f..994284019f 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -295,6 +295,11 @@ struct ParseState
* of SQL:2008 requires us to do it this way. We also use p_lateral_ok to
* forbid LATERAL references to an UPDATE/DELETE target table.
*
+ * While processing the RETURNING clause, special namespace items are added to
+ * refer to the OLD and NEW state of the result relation. These namespace
+ * items have p_returning_type set appropriately, for use when creating Vars.
+ * For convenience, this information is duplicated on each namespace column.
+ *
* At no time should a namespace list contain two entries that conflict
* according to the rules in checkNameSpaceConflicts; but note that those
* are more complicated than "must have different alias names", so in practice
@@ -312,6 +317,7 @@ struct ParseNamespaceItem
bool p_cols_visible; /* Column names visible as unqualified refs? */
bool p_lateral_only; /* Is only visible to LATERAL expressions? */
bool p_lateral_ok; /* If so, does join type allow use? */
+ VarReturningType p_returning_type; /* Is OLD/NEW for use in RETURNING? */
};
/*
@@ -342,6 +348,7 @@ struct ParseNamespaceColumn
Oid p_vartype; /* pg_type OID */
int32 p_vartypmod; /* type modifier value */
Oid p_varcollid; /* OID of collation, or InvalidOid */
+ VarReturningType p_varreturningtype; /* for RETURNING OLD/NEW */
Index p_varnosyn; /* rangetable index of syntactic referent */
AttrNumber p_varattnosyn; /* attribute number of syntactic referent */
bool p_dontexpand; /* not included in star expansion */
diff --git a/src/include/parser/parse_relation.h b/src/include/parser/parse_relation.h
index df6fd5550d..3ece5cd4ee 100644
--- a/src/include/parser/parse_relation.h
+++ b/src/include/parser/parse_relation.h
@@ -114,6 +114,7 @@ extern void errorMissingRTE(ParseState *pstate, RangeVar *relation) pg_attribute
extern void errorMissingColumn(ParseState *pstate,
const char *relname, const char *colname, int location) pg_attribute_noreturn();
extern void expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
+ VarReturningType returning_type,
int location, bool include_dropped,
List **colnames, List **colvars);
extern List *expandNSItemVars(ParseState *pstate, ParseNamespaceItem *nsitem,
diff --git a/src/include/rewrite/rewriteManip.h b/src/include/rewrite/rewriteManip.h
index 1070b93a9d..512823033b 100644
--- a/src/include/rewrite/rewriteManip.h
+++ b/src/include/rewrite/rewriteManip.h
@@ -89,6 +89,7 @@ extern Node *ReplaceVarsFromTargetList(Node *node,
int target_varno, int sublevels_up,
RangeTblEntry *target_rte,
List *targetlist,
+ int result_relation,
ReplaceVarsNoMatchOption nomatch_option,
int nomatch_varno,
bool *outer_hasSubLinks);
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
index ad9aec63cb..f22ca213c2 100644
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -105,8 +105,8 @@ my %replace_line = (
'SHOW TRANSACTION ISOLATION LEVEL ecpg_into',
'VariableShowStmt SHOW SESSION AUTHORIZATION' =>
'SHOW SESSION AUTHORIZATION ecpg_into',
- 'returning_clause RETURNING target_list' =>
- 'RETURNING target_list opt_ecpg_into',
+ 'returning_clause RETURNING returning_with_clause target_list' =>
+ 'RETURNING returning_with_clause target_list opt_ecpg_into',
'ExecuteStmt EXECUTE name execute_param_clause' =>
'EXECUTE prepared_name execute_param_clause execute_rest',
'ExecuteStmt CREATE OptTemp TABLE create_as_target AS EXECUTE name execute_param_clause opt_with_data'
diff --git a/src/test/isolation/expected/merge-update.out b/src/test/isolation/expected/merge-update.out
index 3063c0c6ab..677263d1ec 100644
--- a/src/test/isolation/expected/merge-update.out
+++ b/src/test/isolation/expected/merge-update.out
@@ -40,12 +40,12 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -98,14 +98,14 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+------------------------------------------------------
-UPDATE | 3|setup1 updated by merge1 source not matched by merge2a
-INSERT | 1|merge2a
+merge_action|old |new |key|val
+------------+------------------------------+------------------------------------------------------------+---+------------------------------------------------------
+UPDATE |(2,"setup1 updated by merge1")|(3,"setup1 updated by merge1 source not matched by merge2a")| 3|setup1 updated by merge1 source not matched by merge2a
+INSERT | |(1,merge2a) | 1|merge2a
(2 rows)
step select2: SELECT * FROM target;
@@ -137,13 +137,13 @@ step merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step a1: ABORT;
step merge2a: <... completed>
-merge_action|key|val
-------------+---+-------------------------
-UPDATE | 2|setup1 updated by merge2a
+merge_action|old |new |key|val
+------------+----------+-------------------------------+---+-------------------------
+UPDATE |(1,setup1)|(2,"setup1 updated by merge2a")| 2|setup1 updated by merge2a
(1 row)
step select2: SELECT * FROM target;
@@ -234,14 +234,14 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
-merge_action|key|val
-------------+---+--------------------------------------------------
-UPDATE | 2|initial updated by pa_merge1 updated by pa_merge2a
-UPDATE | 3|initial source not matched by pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+--------------------------------------------------------+---+--------------------------------------------------
+UPDATE |(1,"initial updated by pa_merge1")|(2,"initial updated by pa_merge1 updated by pa_merge2a")| 2|initial updated by pa_merge1 updated by pa_merge2a
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
(2 rows)
step pa_select2: SELECT * FROM pa_target;
@@ -273,7 +273,7 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
<waiting ...>
step c1: COMMIT;
step pa_merge2a: <... completed>
@@ -303,13 +303,13 @@ step pa_merge2a:
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
-merge_action|key|val
-------------+---+-------------------------------------------------------------
-UPDATE | 3|initial source not matched by pa_merge2a
-UPDATE | 3|initial updated by pa_merge2 source not matched by pa_merge2a
-INSERT | 1|pa_merge2a
+merge_action|old |new |key|val
+------------+----------------------------------+-------------------------------------------------------------------+---+-------------------------------------------------------------
+UPDATE |(2,initial) |(3,"initial source not matched by pa_merge2a") | 3|initial source not matched by pa_merge2a
+UPDATE |(2,"initial updated by pa_merge2")|(3,"initial updated by pa_merge2 source not matched by pa_merge2a")| 3|initial updated by pa_merge2 source not matched by pa_merge2a
+INSERT | |(1,pa_merge2a) | 1|pa_merge2a
(3 rows)
step pa_select2: SELECT * FROM pa_target;
diff --git a/src/test/isolation/specs/merge-update.spec b/src/test/isolation/specs/merge-update.spec
index a33dcdba53..c718ff646b 100644
--- a/src/test/isolation/specs/merge-update.spec
+++ b/src/test/isolation/specs/merge-update.spec
@@ -95,7 +95,7 @@ step "merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
step "merge2b"
{
@@ -128,7 +128,7 @@ step "pa_merge2a"
UPDATE set key = t.key + 1, val = t.val || ' updated by ' || s.val
WHEN NOT MATCHED BY SOURCE THEN
UPDATE set key = t.key + 1, val = t.val || ' source not matched by pa_merge2a'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
}
# MERGE proceeds only if 'val' unchanged
step "pa_merge2b_when"
diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out
index 28d8551063..05314ad439 100644
--- a/src/test/regress/expected/merge.out
+++ b/src/test/regress/expected/merge.out
@@ -297,13 +297,13 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
- merge_action | tid | balance
---------------+-----+---------
- DELETE | 1 | 10
- DELETE | 2 | 20
- DELETE | 3 | 30
- INSERT | 4 | 40
+RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ DELETE | (1,10) | | 1 | 10
+ DELETE | (2,20) | | 2 | 20
+ DELETE | (3,30) | | 3 | 30
+ INSERT | | (4,40) | 4 | 40
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -994,7 +994,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 THEN
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
NOTICE: BEFORE INSERT STATEMENT trigger
NOTICE: BEFORE UPDATE STATEMENT trigger
NOTICE: BEFORE DELETE STATEMENT trigger
@@ -1009,12 +1009,12 @@ NOTICE: AFTER UPDATE ROW trigger row: (1,10) -> (1,0)
NOTICE: AFTER DELETE STATEMENT trigger
NOTICE: AFTER UPDATE STATEMENT trigger
NOTICE: AFTER INSERT STATEMENT trigger
- merge_action | tid | balance
---------------+-----+---------
- UPDATE | 3 | 10
- INSERT | 4 | 40
- DELETE | 2 | 20
- UPDATE | 1 | 0
+ merge_action | old | new | tid | balance
+--------------+--------+--------+-----+---------
+ UPDATE | (3,30) | (3,10) | 3 | 10
+ INSERT | | (4,40) | 4 | 40
+ DELETE | (2,20) | | 2 | 20
+ UPDATE | (1,10) | (1,0) | 1 | 0
(4 rows)
SELECT * FROM target ORDER BY tid;
@@ -1436,17 +1436,19 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
WHEN 'DELETE' THEN 'Removed '||t
END AS description;
- action | tid | balance | description
---------+-----+---------+---------------------
- del | 1 | 100 | Removed (1,100)
- upd | 2 | 220 | Added 20 to balance
- ins | 4 | 40 | Inserted (4,40)
+ action | old_tid | old_balance | new_tid | new_balance | delta_balance | tid | balance | description
+--------+---------+-------------+---------+-------------+---------------+-----+---------+---------------------
+ del | 1 | 100 | | | | 1 | 100 | Removed (1,100)
+ upd | 2 | 200 | 2 | 220 | 20 | 2 | 220 | Added 20 to balance
+ ins | | | 4 | 40 | | 4 | 40 | Inserted (4,40)
(3 rows)
ROLLBACK;
@@ -1473,7 +1475,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -1487,14 +1489,14 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
- action | log_action | tid | last_change
---------+------------+-----+---------------------
- DELETE | UPDATE | 1 | Removed (1,100)
- UPDATE | INSERT | 2 | Added 20 to balance
- INSERT | INSERT | 4 | Inserted (4,40)
+ action | old_data | new_data | tid | balance | description | log_action | old_log | new_log | tid | last_change
+--------+----------+----------+-----+---------+---------------------+------------+----------------------+---------------------------+-----+---------------------
+ DELETE | (1,100) | | 1 | 100 | Removed (1,100) | UPDATE | (1,"Original value") | (1,"Removed (1,100)") | 1 | Removed (1,100)
+ UPDATE | (2,200) | (2,220) | 2 | 220 | Added 20 to balance | INSERT | | (2,"Added 20 to balance") | 2 | Added 20 to balance
+ INSERT | | (4,40) | 4 | 40 | Inserted (4,40) | INSERT | | (4,"Inserted (4,40)") | 4 | Inserted (4,40)
(3 rows)
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -1518,11 +1520,11 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
-DELETE 1 100
-UPDATE 2 220
-INSERT 4 40
+DELETE 1 100 \N \N
+UPDATE 2 200 2 220
+INSERT \N \N 4 40
ROLLBACK;
-- SQL function with MERGE ... RETURNING
BEGIN;
@@ -2039,10 +2041,10 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
- merge_action | tid | balance | val
---------------+-----+---------+--------------------------
- UPDATE | 2 | 110 | initial updated by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | tid | balance | val
+--------------+-----------------+------------------------------------+-----+---------+--------------------------
+ UPDATE | (1,100,initial) | (2,110,"initial updated by merge") | 2 | 110 | initial updated by merge
(1 row)
SELECT * FROM pa_target ORDER BY tid;
@@ -2324,18 +2326,18 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
- merge_action | logts | tid | balance | val
---------------+--------------------------+-----+---------+--------------------------
- UPDATE | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
- UPDATE | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
- UPDATE | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
- INSERT | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
+ RETURNING merge_action(), old, new, t.*;
+ merge_action | old | new | logts | tid | balance | val
+--------------+--------------------------------------------+---------------------------------------------------------------+--------------------------+-----+---------+--------------------------
+ UPDATE | ("Tue Jan 31 00:00:00 2017",1,100,initial) | ("Tue Jan 31 00:00:00 2017",1,110,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 1 | 110 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",2,200,initial) | ("Tue Feb 28 00:00:00 2017",2,220,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 2 | 220 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",3,30,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 3 | 30 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",4,400,initial) | ("Tue Jan 31 00:00:00 2017",4,440,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 4 | 440 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",5,500,initial) | ("Tue Feb 28 00:00:00 2017",5,550,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 5 | 550 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",6,60,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 6 | 60 | inserted by merge
+ UPDATE | ("Tue Jan 31 00:00:00 2017",7,700,initial) | ("Tue Jan 31 00:00:00 2017",7,770,"initial updated by merge") | Tue Jan 31 00:00:00 2017 | 7 | 770 | initial updated by merge
+ UPDATE | ("Tue Feb 28 00:00:00 2017",8,800,initial) | ("Tue Feb 28 00:00:00 2017",8,880,"initial updated by merge") | Tue Feb 28 00:00:00 2017 | 8 | 880 | initial updated by merge
+ INSERT | | ("Sun Jan 15 00:00:00 2017",9,90,"inserted by merge") | Sun Jan 15 00:00:00 2017 | 9 | 90 | inserted by merge
(9 rows)
SELECT * FROM pa_target ORDER BY tid;
diff --git a/src/test/regress/expected/returning.out b/src/test/regress/expected/returning.out
index cb51bb8687..9316ddd48a 100644
--- a/src/test/regress/expected/returning.out
+++ b/src/test/regress/expected/returning.out
@@ -355,3 +355,551 @@ INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.f3; -- ok
42
(1 row)
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+ERROR: syntax error at or near "nonsuch"
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS so...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+ERROR: table name "foo" specified more than once
+LINE 1: INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+ERROR: OLD cannot be specified multiple times
+LINE 1: ...EFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) ...
+ ^
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, new AS n) *;
+ERROR: NEW cannot be specified multiple times
+LINE 1: ...EFAULT VALUES RETURNING WITH (old AS o, new AS n, new AS n) ...
+ ^
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ -> Result
+ Output: 4, NULL::text, 42, '99'::bigint
+(4 rows)
+
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+------+----+----+----+----+----------+-------+----+----+----+----+----+----+----+----
+ | | | | | | foo | (0,4) | 4 | | 42 | 99 | 4 | | 42 | 99
+(1 row)
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ QUERY PLAN
+----------------------------------------------------------------------------------------------------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (o.tableoid)::regclass, o.ctid, o.f1, o.f2, o.f3, o.f4, (n.tableoid)::regclass, n.ctid, n.f1, n.f2, n.f3, n.f4, foo.f1, foo.f2, foo.f3, foo.f4
+ Conflict Resolution: UPDATE
+ Conflict Arbiter Indexes: foo_f1_idx
+ -> Values Scan on "*VALUES*"
+ Output: "*VALUES*".column1, "*VALUES*".column2, 42, '99'::bigint
+(6 rows)
+
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+----+----------+-------+----+------------+----+----+----+------------+----+----
+ foo | (0,4) | 4 | | 42 | 99 | foo | (0,5) | 4 | conflicted | -1 | 99 | 4 | conflicted | -1 | 99
+ | | | | | | foo | (0,6) | 5 | ok | 42 | 99 | 5 | ok | 42 | 99
+(2 rows)
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ QUERY PLAN
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, old.*, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, new.*, (((old.f4)::text || '->'::text) || (new.f4)::text)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(8 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+ tableoid | ctid | f1 | f2 | f3 | f4 | old | tableoid | ctid | f1 | f2 | f3 | f4 | new | change
+----------+-------+----+----+----+----+--------------+----------+-------+----+----+----+-----+---------------+---------
+ foo | (0,6) | 5 | ok | 42 | 99 | (5,ok,42,99) | foo | (0,7) | 5 | ok | 42 | 100 | (5,ok,42,100) | 99->100
+(1 row)
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ QUERY PLAN
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (old.tableoid)::regclass, old.ctid, old.f1, old.f2, old.f3, old.f4, (new.tableoid)::regclass, new.ctid, new.f1, new.f2, new.f3, new.f4, foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+(6 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | f1 | f2 | f3 | f4 | tableoid | ctid | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----------+-------+----+----+----+-----+----------+------+----+----+----+----+----+----+----+-----
+ foo | (0,7) | 5 | ok | 42 | 100 | | | | | | | 5 | ok | 42 | 100
+(1 row)
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Insert on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ -> Result
+ Output: 5, 'subquery test'::text, 42, '99'::bigint
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(16 rows)
+
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ | 109
+(1 row)
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2), (SubPlan 3)
+ Update on pg_temp.foo foo_1
+ -> Result
+ Output: '100'::bigint, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Result
+ Output: (old.f4 = new.f4)
+ SubPlan 2
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 3
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(23 rows)
+
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ ?column? | old_max | new_max
+----------+---------+---------
+ f | 109 | 110
+(1 row)
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ QUERY PLAN
+---------------------------------------------------------------
+ Delete on pg_temp.foo
+ Output: (SubPlan 1), (SubPlan 2)
+ Delete on pg_temp.foo foo_1
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.tableoid, foo_1.ctid
+ Filter: (foo_1.f1 = 5)
+ SubPlan 1
+ -> Aggregate
+ Output: max((old.f4 + x.x))
+ -> Function Scan on pg_catalog.generate_series x
+ Output: x.x
+ Function Call: generate_series(1, 10)
+ SubPlan 2
+ -> Aggregate
+ Output: max((new.f4 + x_1.x))
+ -> Function Scan on pg_catalog.generate_series x_1
+ Output: x_1.x
+ Function Call: generate_series(1, 10)
+(18 rows)
+
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+ old_max | new_max
+---------+---------
+ 110 |
+(1 row)
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, new.f1, new.f2, new.f3, new.f4, foo_2.f1, foo_2.f2, foo_2.f3, foo_2.f4
+ Update on pg_temp.foo foo_2
+ -> Nested Loop
+ Output: (foo_2.f2 || ' (deleted)'::text), '-1'::integer, '-1'::bigint, foo_1.ctid, foo_1.tableoid, foo_2.tableoid, foo_2.ctid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f2, foo_2.f1, foo_2.tableoid, foo_2.ctid
+ Filter: (foo_2.f1 = 4)
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.ctid, foo_1.f1, foo_1.tableoid
+ Filter: (foo_1.f1 = 4)
+(11 rows)
+
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+ f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4 | f1 | f2 | f3 | f4
+----+------------+----+----+----+----------------------+----+----+----+----------------------+----+----
+ 4 | conflicted | -1 | 99 | 4 | conflicted (deleted) | -1 | -1 | 4 | conflicted (deleted) | -1 | -1
+(1 row)
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.foo
+ Output: old.f1, old.f2, old.f3, old.f4, joinme.other, new.f1, new.f2, new.f3, new.f4, joinme.other, foo_1.f1, foo_1.f2, foo_1.f3, foo_1.f4, joinme.other, (new.f3 - old.f3)
+ Update on pg_temp.foo foo_1
+ -> Hash Join
+ Output: foo_2.f1, (foo_2.f3 + 1), joinme.ctid, foo_2.ctid, joinme_1.ctid, joinme.other, foo_1.tableoid, foo_1.ctid, foo_2.tableoid
+ Hash Cond: (foo_1.f2 = joinme.f2j)
+ -> Hash Join
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme_1.ctid, joinme_1.f2j
+ Hash Cond: (joinme_1.f2j = foo_1.f2)
+ -> Seq Scan on pg_temp.joinme joinme_1
+ Output: joinme_1.ctid, joinme_1.f2j
+ -> Hash
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Seq Scan on pg_temp.foo foo_1
+ Output: foo_1.f2, foo_1.tableoid, foo_1.ctid
+ -> Hash
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Hash Join
+ Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Hash Cond: (joinme.f2j = foo_2.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.ctid, joinme.other, joinme.f2j
+ -> Hash
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ -> Seq Scan on pg_temp.foo foo_2
+ Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid
+ Filter: (foo_2.f3 = 57)
+(27 rows)
+
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 57 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 58 | 99 | 54321 | 1
+(1 row)
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ERROR: column reference "new.f1" is ambiguous
+LINE 3: RETURNING new.f1, new.f4
+ ^
+DETAIL: It could refer to either a PL/pgSQL variable or a table column.
+QUERY: UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4
+CONTEXT: PL/pgSQL function joinview_upd_trig_fn() line 4 at SQL statement
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Update on pg_temp.joinview
+ Output: old.f1, old.f2, old.f3, old.f4, old.other, new.f1, new.f2, new.f3, new.f4, new.other, joinview.f1, joinview.f2, joinview.f3, joinview.f4, joinview.other, (new.f3 - old.f3)
+ -> Hash Join
+ Output: (foo.f3 + 1), '7'::bigint, ROW(foo.f1, foo.f2, foo.f3, foo.f4, joinme.other), foo.ctid, joinme.ctid, foo.tableoid
+ Hash Cond: (joinme.f2j = foo.f2)
+ -> Seq Scan on pg_temp.joinme
+ Output: joinme.other, joinme.ctid, joinme.f2j
+ -> Hash
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ -> Seq Scan on pg_temp.foo
+ Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid
+ Filter: (foo.f3 = 58)
+(12 rows)
+
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+NOTICE: UPDATE: (3,zoo2,58,99,54321) -> (3,zoo2,59,7,54321)
+ f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | f1 | f2 | f3 | f4 | other | delta_f3
+----+------+----+----+-------+----+------+----+----+-------+----+------+----+----+-------+----------
+ 3 | zoo2 | 58 | 99 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 3 | zoo2 | 59 | 70 | 54321 | 1
+(1 row)
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+NOTICE: drop cascades to 3 other objects
+DETAIL: drop cascades to rule voo_i on view voo
+drop cascades to view joinview
+drop cascades to rule foo_del_rule on table foo
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+ERROR: column old.f3 does not exist
+LINE 1: UPDATE foo SET f4 = f4 + 1 RETURNING old.f3;
+ ^
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+ old | new
+-------------------------------+------------------------------
+ (1,xxx,20) | (1,xxx,21)
+ (2,more,141) | (2,more,142)
+ (4,"conflicted (deleted)",-1) | (4,"conflicted (deleted)",0)
+ (3,zoo2,70) | (3,zoo2,71)
+(4 rows)
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT RETURNING old.*, new.*, *;
+ERROR: RETURNING must have at least one column
+LINE 1: INSERT INTO zerocol SELECT RETURNING old.*, new.*, *;
+ ^
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+------+----------+-------+-------
+ | | zerocol | (0,1) | (0,1)
+(1 row)
+
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+ tableoid | ctid | tableoid | ctid | ctid
+----------+-------+----------+------+-------
+ zerocol | (0,1) | | | (0,1)
+(1 row)
+
+DROP TABLE zerocol;
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+----------+------+---+---+---+-------------+-------+---+------+----+---+------+----
+ | | | | | foo_part_s1 | (0,1) | 1 | 17.1 | P1 | 1 | 17.1 | P1
+ | | | | | foo_part_s2 | (0,1) | 2 | 17.2 | P2 | 2 | 17.2 | P2
+ | | | | | foo_part_d1 | (0,1) | 3 | 17.3 | P3 | 3 | 17.3 | P3
+ | | | | | foo_part_d2 | (0,1) | 4 | 17.4 | P4 | 4 | 17.4 | P4
+(4 rows)
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_s1 | (0,1) | 1 | 17.1 | P1 | foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | 2 | 18.1 | P1->P2
+(1 row)
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----+-------------+-------+---+------+--------+---+------+--------
+ foo_part_d1 | (0,1) | 3 | 17.3 | P3 | foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | 1 | 18.3 | P3->P1
+(1 row)
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+--------+-------------+-------+---+------+------------+---+------+------------
+ foo_part_s1 | (0,2) | 1 | 18.3 | P3->P1 | foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | 3 | 19.3 | P3->P1->P3
+(1 row)
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+------------+-------------+-------+---+------+----------------+---+------+----------------
+ foo_part_d1 | (0,2) | 3 | 19.3 | P3->P1->P3 | foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | 4 | 20.3 | P3->P1->P3->P4
+(1 row)
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+ tableoid | ctid | a | b | c | tableoid | ctid | a | b | c | a | b | c
+-------------+-------+---+------+----------------+----------+------+---+---+---+---+------+----------------
+ foo_part_s2 | (0,1) | 2 | 17.2 | P2 | | | | | | 2 | 17.2 | P2
+ foo_part_s2 | (0,2) | 2 | 18.1 | P1->P2 | | | | | | 2 | 18.1 | P1->P2
+ foo_part_d2 | (0,1) | 4 | 17.4 | P4 | | | | | | 4 | 17.4 | P4
+ foo_part_d2 | (0,2) | 4 | 20.3 | P3->P1->P3->P4 | | | | | | 4 | 20.3 | P3->P1->P3->P4
+(4 rows)
+
+DROP TABLE foo_parted;
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ WITH u1 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING old.*, new.*
+ ), u2 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (OLD AS "old foo") "old foo".*, new.*
+ ), u3 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (NEW AS "new foo") old.*, "new foo".*
+ )
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.*, n.*, o, n, o.f1 = n.f1, o = n,
+ (SELECT o.f2 = n.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = n.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = n);
+END;
+\sf foo_update
+CREATE OR REPLACE FUNCTION public.foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ WITH u1 AS (
+ UPDATE foo foo_1 SET f1 = (foo_1.f1 + 1)
+ RETURNING old.f1,
+ old.f2,
+ old.f4,
+ new.f1,
+ new.f2,
+ new.f4
+ ), u2 AS (
+ UPDATE foo foo_1 SET f1 = (foo_1.f1 + 1)
+ RETURNING WITH (OLD AS "old foo") "old foo".f1,
+ "old foo".f2,
+ "old foo".f4,
+ new.f1,
+ new.f2,
+ new.f4
+ ), u3 AS (
+ UPDATE foo foo_1 SET f1 = (foo_1.f1 + 1)
+ RETURNING WITH (NEW AS "new foo") old.f1,
+ old.f2,
+ old.f4,
+ "new foo".f1,
+ "new foo".f2,
+ "new foo".f4
+ )
+ UPDATE foo SET f1 = (foo.f1 + 1)
+ RETURNING WITH (OLD AS o, NEW AS n) o.f1,
+ o.f2,
+ o.f4,
+ n.f1,
+ n.f2,
+ n.f4,
+ o.*::foo AS o,
+ n.*::foo AS n,
+ (o.f1 = n.f1),
+ (o.* = n.*),
+ ( SELECT (o.f2 = n.f2)),
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f1 = o.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.f4 = n.f4)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = o.*)) AS count,
+ ( SELECT count(*) AS count
+ FROM foo foo_1
+ WHERE (foo_1.* = n.*)) AS count;
+END
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 3014d047fe..76ed380247 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -3647,7 +3647,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s
-- test deparsing
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3686,11 +3689,12 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
CREATE OR REPLACE FUNCTION public.merge_sf_test()
- RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[])
+ RETURNS TABLE(action text, a integer, b text, id integer, data text, filling integer[], old_id integer, old_data text, old_filling integer[], new_id integer, new_data text, new_filling integer[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -3728,12 +3732,18 @@ BEGIN ATOMIC
WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
- RETURNING MERGE_ACTION() AS action,
+ RETURNING WITH (OLD AS o, NEW AS n) MERGE_ACTION() AS action,
s.a,
s.b,
t.id,
t.data,
- t.filling;
+ t.filling,
+ o.id,
+ o.data,
+ o.filling,
+ n.id,
+ n.data,
+ n.filling;
END
CREATE FUNCTION merge_sf_test2()
RETURNS void
diff --git a/src/test/regress/expected/updatable_views.out b/src/test/regress/expected/updatable_views.out
index 8786058ed0..095df0a670 100644
--- a/src/test/regress/expected/updatable_views.out
+++ b/src/test/regress/expected/updatable_views.out
@@ -437,7 +437,8 @@ NOTICE: drop cascades to view ro_view19
-- simple updatable view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const' AS c, (SELECT concat('b: ', b)) AS d FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view1';
@@ -462,7 +463,9 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | YES
rw_view1 | b | YES
-(2 rows)
+ rw_view1 | c | NO
+ rw_view1 | d | NO
+(4 rows)
INSERT INTO rw_view1 VALUES (3, 'Row 3');
INSERT INTO rw_view1 (a) VALUES (4);
@@ -479,20 +482,22 @@ SELECT * FROM base_tbl;
5 | Unspecified
(6 rows)
+SET jit_above_cost = 0;
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+-------+---+-------------
- UPDATE | 1 | ROW 1 | 1 | ROW 1
- DELETE | 3 | ROW 3 | 3 | Row 3
- INSERT | 2 | ROW 2 | 2 | Unspecified
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
+ merge_action | a | b | old | new | a | b | c | d | a | b | c | d | a | b | c | d
+--------------+---+-------+------------------------------+----------------------------------------+---+-------+-------+----------+---+-------------+-------+----------------+---+-------------+-------+----------------
+ UPDATE | 1 | ROW 1 | (1,"Row 1",Const,"b: Row 1") | (1,"ROW 1",Const,"b: ROW 1") | 1 | Row 1 | Const | b: Row 1 | 1 | ROW 1 | Const | b: ROW 1 | 1 | ROW 1 | Const | b: ROW 1
+ DELETE | 3 | ROW 3 | (3,"Row 3",Const,"b: Row 3") | | 3 | Row 3 | Const | b: Row 3 | | | | | 3 | Row 3 | Const | b: Row 3
+ INSERT | 2 | ROW 2 | | (2,Unspecified,Const,"b: Unspecified") | | | | | 2 | Unspecified | Const | b: Unspecified | 2 | Unspecified | Const | b: Unspecified
(3 rows)
+SET jit_above_cost TO DEFAULT;
SELECT * FROM base_tbl ORDER BY a;
a | b
----+-------------
@@ -511,13 +516,13 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------------
- UPDATE | 1 | R1 | 1 | R1
- DELETE | | | 5 | Unspecified
- DELETE | 2 | R2 | 2 | Unspecified
- INSERT | 3 | R3 | 3 | Unspecified
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
+ merge_action | a | b | old | new | a | b | c | d | a | b | c | d | a | b | c | d
+--------------+---+----+----------------------------------------+----------------------------------------+---+-------------+-------+----------------+---+-------------+-------+----------------+---+-------------+-------+----------------
+ UPDATE | 1 | R1 | (1,"ROW 1",Const,"b: ROW 1") | (1,R1,Const,"b: R1") | 1 | ROW 1 | Const | b: ROW 1 | 1 | R1 | Const | b: R1 | 1 | R1 | Const | b: R1
+ DELETE | | | (5,Unspecified,Const,"b: Unspecified") | | 5 | Unspecified | Const | b: Unspecified | | | | | 5 | Unspecified | Const | b: Unspecified
+ DELETE | 2 | R2 | (2,Unspecified,Const,"b: Unspecified") | | 2 | Unspecified | Const | b: Unspecified | | | | | 2 | Unspecified | Const | b: Unspecified
+ INSERT | 3 | R3 | | (3,Unspecified,Const,"b: Unspecified") | | | | | 3 | Unspecified | Const | b: Unspecified | 3 | Unspecified | Const | b: Unspecified
(4 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -634,8 +639,10 @@ DROP TABLE base_tbl_hist;
-- view on top of view
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name = 'rw_view2';
@@ -660,27 +667,29 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view2 | aaa | YES
rw_view2 | bbb | YES
-(2 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(4 rows)
INSERT INTO rw_view2 VALUES (3, 'Row 3');
INSERT INTO rw_view2 (aaa) VALUES (4);
SELECT * FROM rw_view2;
- aaa | bbb
------+-------------
- 1 | Row 1
- 2 | Row 2
- 3 | Row 3
- 4 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Unspecified | Const1 | Const2
(4 rows)
UPDATE rw_view2 SET bbb='Row 4' WHERE aaa=4;
DELETE FROM rw_view2 WHERE aaa=2;
SELECT * FROM rw_view2;
- aaa | bbb
------+-------
- 1 | Row 1
- 3 | Row 3
- 4 | Row 4
+ aaa | bbb | c1 | c2
+-----+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 3 | Row 3 | Const1 | Const2
+ 4 | Row 4 | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -688,20 +697,20 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-------------
- DELETE | 3 | R3 | 3 | Row 3
- UPDATE | 4 | R4 | 4 | R4
- INSERT | 5 | R5 | 5 | Unspecified
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+---------------------------+-------------------------------+-----+-------------+--------+--------
+ DELETE | 3 | R3 | (3,"Row 3",Const1,Const2) | | 3 | Row 3 | Const1 | Const2
+ UPDATE | 4 | R4 | (4,"Row 4",Const1,Const2) | (4,R4,Const1,Const2) | 4 | R4 | Const1 | Const2
+ INSERT | 5 | R5 | | (5,Unspecified,Const1,Const2) | 5 | Unspecified | Const1 | Const2
(3 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-------------
- 1 | Row 1
- 4 | R4
- 5 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-------------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 4 | R4 | Const1 | Const2
+ 5 | Unspecified | Const1 | Const2
(3 rows)
MERGE INTO rw_view2 t
@@ -710,21 +719,21 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
- merge_action | a | b | aaa | bbb
---------------+---+----+-----+-----------------------
- UPDATE | | | 1 | Not matched by source
- DELETE | 4 | r4 | 4 | R4
- UPDATE | 5 | r5 | 5 | r5
- INSERT | 6 | r6 | 6 | Unspecified
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
+ merge_action | a | b | old | new | aaa | bbb | c1 | c2
+--------------+---+----+-------------------------------+-------------------------------------------+-----+-----------------------+--------+--------
+ UPDATE | | | (1,"Row 1",Const1,Const2) | (1,"Not matched by source",Const1,Const2) | 1 | Not matched by source | Const1 | Const2
+ DELETE | 4 | r4 | (4,R4,Const1,Const2) | | 4 | R4 | Const1 | Const2
+ UPDATE | 5 | r5 | (5,Unspecified,Const1,Const2) | (5,r5,Const1,Const2) | 5 | r5 | Const1 | Const2
+ INSERT | 6 | r6 | | (6,Unspecified,Const1,Const2) | 6 | Unspecified | Const1 | Const2
(4 rows)
SELECT * FROM rw_view2 ORDER BY aaa;
- aaa | bbb
------+-----------------------
- 1 | Not matched by source
- 5 | r5
- 6 | Unspecified
+ aaa | bbb | c1 | c2
+-----+-----------------------+--------+--------
+ 1 | Not matched by source | Const1 | Const2
+ 5 | r5 | Const1 | Const2
+ 6 | Unspecified | Const1 | Const2
(3 rows)
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -886,16 +895,25 @@ SELECT table_name, column_name, is_updatable
rw_view2 | b | YES
(4 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | a | b
+---+---+---+-------
+ | | 3 | Row 3
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+ a | b | a | b
+---+----+---+----
+ 3 | R3 | 3 | R3
+(1 row)
+
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+----+---+-----------
+ 3 | R3 | 3 | Row three
(1 row)
SELECT * FROM rw_view2;
@@ -906,10 +924,10 @@ SELECT * FROM rw_view2;
3 | Row three
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | a | b
+---+-----------+---+---
+ 3 | Row three | |
(1 row)
SELECT * FROM rw_view2;
@@ -960,8 +978,10 @@ drop cascades to view rw_view2
-- view on top of view with triggers
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
WHERE table_name LIKE 'rw_view%'
@@ -992,9 +1012,12 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE FUNCTION rw_view1_trig_fn()
RETURNS trigger AS
@@ -1002,9 +1025,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -1045,9 +1070,12 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_upd_trig INSTEAD OF UPDATE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1081,9 +1109,12 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
CREATE TRIGGER rw_view1_del_trig INSTEAD OF DELETE ON rw_view1
FOR EACH ROW EXECUTE PROCEDURE rw_view1_trig_fn();
@@ -1117,41 +1148,44 @@ SELECT table_name, column_name, is_updatable
------------+-------------+--------------
rw_view1 | a | NO
rw_view1 | b | NO
+ rw_view1 | c1 | NO
rw_view2 | a | NO
rw_view2 | b | NO
-(4 rows)
+ rw_view2 | c1 | NO
+ rw_view2 | c2 | NO
+(7 rows)
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
- a | b
----+-------
- 3 | Row 3
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+---+----+----+---+-------+----------------+--------
+ | | | | 3 | Row 3 | Trigger Const1 | Const2
(1 row)
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-------+--------+--------+---+-----------+----------------+--------
+ 3 | Row 3 | Const1 | Const2 | 3 | Row three | Trigger Const1 | Const2
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-----------
- 1 | Row 1
- 2 | Row 2
- 3 | Row three
+ a | b | c1 | c2
+---+-----------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
+ 3 | Row three | Const1 | Const2
(3 rows)
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
- a | b
----+-----------
- 3 | Row three
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
+ a | b | c1 | c2 | a | b | c1 | c2
+---+-----------+--------+--------+---+---+----+----
+ 3 | Row three | Const1 | Const2 | | | |
(1 row)
SELECT * FROM rw_view2;
- a | b
----+-------
- 1 | Row 1
- 2 | Row 2
+ a | b | c1 | c2
+---+-------+--------+--------
+ 1 | Row 1 | Const1 | Const2
+ 2 | Row 2 | Const1 | Const2
(2 rows)
MERGE INTO rw_view2 t
@@ -1159,12 +1193,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-------
- DELETE | 1 | R1 | 1 | Row 1
- UPDATE | 2 | R2 | 2 | R2
- INSERT | 3 | R3 | 3 | R3
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+---------------------------+--------------------------------+---+-------+----------------+--------
+ DELETE | 1 | R1 | (1,"Row 1",Const1,Const2) | | 1 | Row 1 | Const1 | Const2
+ UPDATE | 2 | R2 | (2,"Row 2",Const1,Const2) | (2,R2,"Trigger Const1",Const2) | 2 | R2 | Trigger Const1 | Const2
+ INSERT | 3 | R3 | | (3,R3,"Trigger Const1",Const2) | 3 | R3 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
@@ -1182,12 +1216,12 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
- merge_action | a | b | a | b
---------------+---+----+---+-----------------------
- UPDATE | 2 | r2 | 2 | r2
- UPDATE | | | 3 | Not matched by source
- INSERT | 1 | r1 | 1 | r1
+ RETURNING merge_action(), s.*, old, new, t.*;
+ merge_action | a | b | old | new | a | b | c1 | c2
+--------------+---+----+----------------------+-----------------------------------------------------+---+-----------------------+----------------+--------
+ UPDATE | 2 | r2 | (2,R2,Const1,Const2) | (2,r2,"Trigger Const1",Const2) | 2 | r2 | Trigger Const1 | Const2
+ UPDATE | | | (3,R3,Const1,Const2) | (3,"Not matched by source","Trigger Const1",Const2) | 3 | Not matched by source | Trigger Const1 | Const2
+ INSERT | 1 | r1 | | (1,r1,"Trigger Const1",Const2) | 1 | r1 | Trigger Const1 | Const2
(3 rows)
SELECT * FROM base_tbl ORDER BY a;
diff --git a/src/test/regress/sql/merge.sql b/src/test/regress/sql/merge.sql
index 54929a92fa..07b6295b3b 100644
--- a/src/test/regress/sql/merge.sql
+++ b/src/test/regress/sql/merge.sql
@@ -235,7 +235,7 @@ WHEN NOT MATCHED BY SOURCE THEN
DELETE
WHEN NOT MATCHED BY TARGET THEN
INSERT VALUES (s.sid, s.delta)
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -677,7 +677,7 @@ WHEN NOT MATCHED BY SOURCE AND tid = 1 THEN
UPDATE SET balance = 0
WHEN NOT MATCHED BY SOURCE THEN
DELETE
-RETURNING merge_action(), t.*;
+RETURNING merge_action(), old, new, t.*;
SELECT * FROM target ORDER BY tid;
ROLLBACK;
@@ -930,7 +930,9 @@ WHEN MATCHED AND tid < 2 THEN
DELETE
RETURNING (SELECT abbrev FROM merge_actions
WHERE action = merge_action()) AS action,
- t.*,
+ old.tid AS old_tid, old.balance AS old_balance,
+ new.tid AS new_tid, new.balance AS new_balance,
+ (SELECT new.balance - old.balance AS delta_balance), t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -956,7 +958,7 @@ WITH m AS (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action() AS action, t.*,
+ RETURNING merge_action() AS action, old AS old_data, new AS new_data, t.*,
CASE merge_action()
WHEN 'INSERT' THEN 'Inserted '||t
WHEN 'UPDATE' THEN 'Added '||delta||' to balance'
@@ -970,7 +972,7 @@ WITH m AS (
UPDATE SET last_change = description
WHEN NOT MATCHED THEN
INSERT VALUES (m.tid, description)
- RETURNING action, merge_action() AS log_action, l.*
+ RETURNING m.*, merge_action() AS log_action, old AS old_log, new AS new_log, l.*
)
SELECT * FROM m2;
SELECT * FROM sq_target_merge_log ORDER BY tid;
@@ -988,7 +990,7 @@ COPY (
INSERT (balance, tid) VALUES (balance + delta, sid)
WHEN MATCHED AND tid < 2 THEN
DELETE
- RETURNING merge_action(), t.*
+ RETURNING merge_action(), old.*, new.*
) TO stdout;
ROLLBACK;
@@ -1265,7 +1267,7 @@ MERGE INTO pa_target t
ON t.tid = s.sid AND t.tid = 1
WHEN MATCHED THEN
UPDATE SET tid = tid + 1, balance = balance + delta, val = val || ' updated by merge'
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
@@ -1456,7 +1458,7 @@ MERGE INTO pa_target t
UPDATE SET balance = balance + delta, val = val || ' updated by merge'
WHEN NOT MATCHED THEN
INSERT VALUES (slogts::timestamp, sid, delta, 'inserted by merge')
- RETURNING merge_action(), t.*;
+ RETURNING merge_action(), old, new, t.*;
SELECT * FROM pa_target ORDER BY tid;
ROLLBACK;
diff --git a/src/test/regress/sql/returning.sql b/src/test/regress/sql/returning.sql
index a460f82fb7..572c7404f7 100644
--- a/src/test/regress/sql/returning.sql
+++ b/src/test/regress/sql/returning.sql
@@ -160,3 +160,214 @@ INSERT INTO foo AS bar DEFAULT VALUES RETURNING *; -- ok
INSERT INTO foo AS bar DEFAULT VALUES RETURNING foo.*; -- fails, wrong name
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.*; -- ok
INSERT INTO foo AS bar DEFAULT VALUES RETURNING bar.f3; -- ok
+
+--
+-- Test RETURNING OLD/NEW.
+--
+-- Start with new data, to ensure predictable TIDs.
+--
+TRUNCATE foo;
+INSERT INTO foo VALUES (1, 'xxx', 10, 20), (2, 'more', 42, 141), (3, 'zoo2', 57, 99);
+
+-- Error cases
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (nonsuch AS something) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (new AS foo) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, old AS o) *;
+INSERT INTO foo DEFAULT VALUES RETURNING WITH (old AS o, new AS n, new AS n) *;
+
+-- INSERT has NEW, but not OLD
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+INSERT INTO foo VALUES (4)
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- INSERT ... ON CONFLICT ... UPDATE has OLD and NEW
+CREATE UNIQUE INDEX foo_f1_idx ON foo (f1);
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+INSERT INTO foo VALUES (4, 'conflict'), (5, 'ok')
+ ON CONFLICT (f1) DO UPDATE SET f2 = excluded.f2||'ed', f3 = -1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.tableoid::regclass, o.ctid, o.*,
+ n.tableoid::regclass, n.ctid, n.*, *;
+
+-- UPDATE has OLD and NEW
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*, old,
+ new.tableoid::regclass, new.ctid, new.*, new,
+ old.f4::text||'->'||new.f4::text AS change;
+
+-- DELETE has OLD, but not NEW
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+-- RETURNING OLD and NEW from subquery
+EXPLAIN (verbose, costs off)
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+INSERT INTO foo VALUES (5, 'subquery test')
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+UPDATE foo SET f4 = 100 WHERE f1 = 5
+ RETURNING (SELECT old.f4 = new.f4),
+ (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+DELETE FROM foo WHERE f1 = 5
+ RETURNING (SELECT max(old.f4 + x) FROM generate_series(1, 10) x) old_max,
+ (SELECT max(new.f4 + x) FROM generate_series(1, 10) x) new_max;
+
+-- DELETE turned into UPDATE by a rule has OLD and NEW
+CREATE RULE foo_del_rule AS ON DELETE TO foo DO INSTEAD
+ UPDATE foo SET f2 = f2||' (deleted)', f3 = -1, f4 = -1 WHERE f1 = OLD.f1
+ RETURNING *;
+EXPLAIN (verbose, costs off)
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+DELETE FROM foo WHERE f1 = 4 RETURNING old.*,new.*, *;
+
+-- UPDATE on view with rule
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+
+-- UPDATE on view with INSTEAD OF trigger
+CREATE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING new.f1, new.f4 INTO new.f1, new.f4; -- should fail
+ RETURN NEW;
+END;
+$$;
+CREATE TRIGGER joinview_upd_trig INSTEAD OF UPDATE ON joinview
+ FOR EACH ROW EXECUTE FUNCTION joinview_upd_trig_fn();
+DROP RULE joinview_u ON joinview;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should fail
+
+CREATE OR REPLACE FUNCTION joinview_upd_trig_fn() RETURNS trigger
+LANGUAGE plpgsql AS
+$$
+BEGIN
+ RAISE NOTICE 'UPDATE: % -> %', old, new;
+ UPDATE foo SET f1 = new.f1, f3 = new.f3, f4 = new.f4 * 10
+ FROM joinme WHERE f2 = f2j AND f2 = old.f2
+ RETURNING WITH (new AS n) new.f1, n.f4 INTO new.f1, new.f4; -- now ok
+ RETURN NEW;
+END;
+$$;
+EXPLAIN (verbose, costs off)
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3;
+UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58
+ RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed
+
+-- Test wholerow & dropped column handling
+ALTER TABLE foo DROP COLUMN f3 CASCADE;
+UPDATE foo SET f4 = f4 + 1 RETURNING old.f3; -- should fail
+UPDATE foo SET f4 = f4 + 1 RETURNING old, new;
+
+-- INSERT/DELETE on zero column table
+CREATE TABLE zerocol();
+INSERT INTO zerocol SELECT RETURNING old.*, new.*, *;
+INSERT INTO zerocol SELECT
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DELETE FROM zerocol
+ RETURNING old.tableoid::regclass, old.ctid,
+ new.tableoid::regclass, new.ctid, ctid, *;
+DROP TABLE zerocol;
+
+-- Test cross-partition updates and attribute mapping
+CREATE TABLE foo_parted (a int, b float8, c text) PARTITION BY LIST (a);
+CREATE TABLE foo_part_s1 PARTITION OF foo_parted FOR VALUES IN (1);
+CREATE TABLE foo_part_s2 PARTITION OF foo_parted FOR VALUES IN (2);
+CREATE TABLE foo_part_d1 (c text, a int, b float8);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d1 FOR VALUES IN (3);
+CREATE TABLE foo_part_d2 (b float8, c text, a int);
+ALTER TABLE foo_parted ATTACH PARTITION foo_part_d2 FOR VALUES IN (4);
+
+INSERT INTO foo_parted
+ VALUES (1, 17.1, 'P1'), (2, 17.2, 'P2'), (3, 17.3, 'P3'), (4, 17.4, 'P4')
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 2, b = b + 1, c = c || '->P2' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 1, b = b + 1, c = c || '->P1' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 3, b = b + 1, c = c || '->P3' WHERE a = 1
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+UPDATE foo_parted SET a = 4, b = b + 1, c = c || '->P4' WHERE a = 3
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DELETE FROM foo_parted
+ RETURNING old.tableoid::regclass, old.ctid, old.*,
+ new.tableoid::regclass, new.ctid, new.*, *;
+
+DROP TABLE foo_parted;
+
+-- Test deparsing
+CREATE FUNCTION foo_update()
+ RETURNS void
+ LANGUAGE sql
+BEGIN ATOMIC
+ WITH u1 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING old.*, new.*
+ ), u2 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (OLD AS "old foo") "old foo".*, new.*
+ ), u3 AS (
+ UPDATE foo SET f1 = f1 + 1 RETURNING WITH (NEW AS "new foo") old.*, "new foo".*
+ )
+ UPDATE foo SET f1 = f1 + 1
+ RETURNING WITH (OLD AS o, NEW AS n)
+ o.*, n.*, o, n, o.f1 = n.f1, o = n,
+ (SELECT o.f2 = n.f2),
+ (SELECT count(*) FROM foo WHERE foo.f1 = o.f4),
+ (SELECT count(*) FROM foo WHERE foo.f4 = n.f4),
+ (SELECT count(*) FROM foo WHERE foo = o),
+ (SELECT count(*) FROM foo WHERE foo = n);
+END;
+
+\sf foo_update
+DROP FUNCTION foo_update;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 4a5fa50585..fdd3ff1d16 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1294,7 +1294,10 @@ MERGE INTO rule_merge1 t USING (SELECT 1 AS a) s
CREATE TABLE sf_target(id int, data text, filling int[]);
CREATE FUNCTION merge_sf_test()
- RETURNS TABLE(action text, a int, b text, id int, data text, filling int[])
+ RETURNS TABLE(action text, a int, b text,
+ id int, data text, filling int[],
+ old_id int, old_data text, old_filling int[],
+ new_id int, new_data text, new_filling int[])
LANGUAGE sql
BEGIN ATOMIC
MERGE INTO sf_target t
@@ -1333,7 +1336,8 @@ WHEN NOT MATCHED
THEN INSERT (filling[1], id)
VALUES (s.a, s.a)
RETURNING
- merge_action() AS action, *;
+ WITH (OLD AS o, NEW AS n)
+ merge_action() AS action, *, o.*, n.*;
END;
\sf merge_sf_test
diff --git a/src/test/regress/sql/updatable_views.sql b/src/test/regress/sql/updatable_views.sql
index 93b693ae83..c071fffc11 100644
--- a/src/test/regress/sql/updatable_views.sql
+++ b/src/test/regress/sql/updatable_views.sql
@@ -154,7 +154,8 @@ DROP SEQUENCE uv_seq CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const' AS c, (SELECT concat('b: ', b)) AS d FROM base_tbl WHERE a>0;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -175,13 +176,18 @@ UPDATE rw_view1 SET a=5 WHERE a=4;
DELETE FROM rw_view1 WHERE b='Row 2';
SELECT * FROM base_tbl;
+SET jit_above_cost = 0;
+
MERGE INTO rw_view1 t
USING (VALUES (0, 'ROW 0'), (1, 'ROW 1'),
(2, 'ROW 2'), (3, 'ROW 3')) AS v(a,b) ON t.a = v.a
WHEN MATCHED AND t.a <= 1 THEN UPDATE SET b = v.b
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
+
+SET jit_above_cost TO DEFAULT;
+
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view1 t
@@ -191,7 +197,7 @@ MERGE INTO rw_view1 t
WHEN MATCHED THEN DELETE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED AND a > 0 THEN INSERT (a) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, new, old.*, new.*, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view1 SET a=6 WHERE a=5;
@@ -240,8 +246,10 @@ DROP TABLE base_tbl_hist;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT b AS bb, a AS aa FROM base_tbl WHERE a>0;
-CREATE VIEW rw_view2 AS SELECT aa AS aaa, bb AS bbb FROM rw_view1 WHERE aa<10;
+CREATE VIEW rw_view1 AS
+ SELECT b AS bb, a AS aa, 'Const1' AS c FROM base_tbl WHERE a>0;
+CREATE VIEW rw_view2 AS
+ SELECT aa AS aaa, bb AS bbb, c AS c1, 'Const2' AS c2 FROM rw_view1 WHERE aa<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -268,7 +276,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND aaa = 3 THEN DELETE
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, (SELECT old), (SELECT (SELECT new)), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
MERGE INTO rw_view2 t
@@ -277,7 +285,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET bbb = v.b
WHEN NOT MATCHED THEN INSERT (aaa) VALUES (v.a)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET bbb = 'Not matched by source'
- RETURNING merge_action(), v.*, t.*;
+ RETURNING merge_action(), v.*, old, (SELECT new FROM (VALUES ((SELECT new)))), t.*;
SELECT * FROM rw_view2 ORDER BY aaa;
EXPLAIN (costs off) UPDATE rw_view2 SET aaa=5 WHERE aaa=4;
@@ -362,10 +370,14 @@ SELECT table_name, column_name, is_updatable
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='R3' WHERE a=3 RETURNING old.*, new.*; -- rule returns NEW
+DROP RULE rw_view1_upd_rule ON rw_view1;
+CREATE RULE rw_view1_upd_rule AS ON UPDATE TO rw_view1
+ DO INSTEAD UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a RETURNING *;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t USING (VALUES (3, 'Row 3')) AS v(a,b) ON t.a = v.a
@@ -381,8 +393,10 @@ DROP TABLE base_tbl CASCADE;
CREATE TABLE base_tbl (a int PRIMARY KEY, b text DEFAULT 'Unspecified');
INSERT INTO base_tbl SELECT i, 'Row ' || i FROM generate_series(-2, 2) g(i);
-CREATE VIEW rw_view1 AS SELECT * FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
-CREATE VIEW rw_view2 AS SELECT * FROM rw_view1 WHERE a<10;
+CREATE VIEW rw_view1 AS
+ SELECT *, 'Const1' AS c1 FROM base_tbl WHERE a>0 OFFSET 0; -- not updatable without rules/triggers
+CREATE VIEW rw_view2 AS
+ SELECT *, 'Const2' AS c2 FROM rw_view1 WHERE a<10;
SELECT table_name, is_insertable_into
FROM information_schema.tables
@@ -407,9 +421,11 @@ $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO base_tbl VALUES (NEW.a, NEW.b);
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
UPDATE base_tbl SET b=NEW.b WHERE a=OLD.a;
+ NEW.c1 = 'Trigger Const1';
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
DELETE FROM base_tbl WHERE a=OLD.a;
@@ -479,10 +495,10 @@ SELECT table_name, column_name, is_updatable
WHERE table_name LIKE 'rw_view%'
ORDER BY table_name, ordinal_position;
-INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING *;
-UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING *;
+INSERT INTO rw_view2 VALUES (3, 'Row 3') RETURNING old.*, new.*;
+UPDATE rw_view2 SET b='Row three' WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
-DELETE FROM rw_view2 WHERE a=3 RETURNING *;
+DELETE FROM rw_view2 WHERE a=3 RETURNING old.*, new.*;
SELECT * FROM rw_view2;
MERGE INTO rw_view2 t
@@ -490,7 +506,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED AND t.a <= 1 THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
MERGE INTO rw_view2 t
@@ -498,7 +514,7 @@ MERGE INTO rw_view2 t
WHEN MATCHED THEN UPDATE SET b = s.b
WHEN NOT MATCHED AND s.a > 0 THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET b = 'Not matched by source'
- RETURNING merge_action(), s.*, t.*;
+ RETURNING merge_action(), s.*, old, new, t.*;
SELECT * FROM base_tbl ORDER BY a;
EXPLAIN (costs off) UPDATE rw_view2 SET a=3 WHERE a=2;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9f83ecf181..15a3a39cfd 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2475,6 +2475,10 @@ RestrictInfo
Result
ResultRelInfo
ResultState
+ReturningClause
+ReturningExpr
+ReturningOption
+ReturningOptionKind
ReturnSetInfo
ReturnStmt
RevmapContents
@@ -2625,6 +2629,7 @@ SetOperationStmt
SetQuantifier
SetToDefault
SetupWorkerPtrType
+SetVarReturningType_context
ShDependObjectInfo
SharedAggInfo
SharedBitmapHeapInstrumentation
@@ -3096,6 +3101,7 @@ VariableSpace
VariableStatData
VariableSubstituteHook
Variables
+VarReturningType
Vector32
Vector8
VersionedQuery
--
2.43.0
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2025-01-16 15:28 ` Dean Rasheed <[email protected]>
2025-01-17 02:23 ` Re: Adding OLD/NEW support to RETURNING Richard Guo <[email protected]>
2025-06-25 13:41 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 2 replies; 87+ messages in thread
From: Dean Rasheed @ 2025-01-16 15:28 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Robert Treat <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Wed, 8 Jan 2025 at 09:38, Dean Rasheed <[email protected]> wrote:
>
> OK, done.
>
> I also noticed that I had failed to use quote_identifier() in
> ruleutils.c for the new WITH aliases, so I've fixed that and adjusted
> a couple of the test cases to test that.
>
I went over this again in detail and didn't find any problems, so I
have committed it. Thanks for all the review comments.
Regards,
Dean
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-16 15:28 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2025-01-17 02:23 ` Richard Guo <[email protected]>
2025-01-17 10:44 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
1 sibling, 1 reply; 87+ messages in thread
From: Richard Guo @ 2025-01-17 02:23 UTC (permalink / raw)
To: Dean Rasheed <[email protected]>; +Cc: jian he <[email protected]>; Robert Treat <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Fri, Jan 17, 2025 at 12:28 AM Dean Rasheed <[email protected]> wrote:
> I went over this again in detail and didn't find any problems, so I
> have committed it. Thanks for all the review comments.
It seems that adding ParseNamespaceItems for the OLD or NEW aliases
may confuse scanNameSpaceForRelid() when searching the table namespace
for a relation item. Since they contain the same RTE,
scanNameSpaceForRelid() might mistakenly detect multiple matches.
create table t (a int, b int);
update public.t set a = 1 returning public.t.b;
ERROR: table reference 46337 is ambiguous
LINE 1: update public.t set a = 1 returning public.t.b;
^
Thanks
Richard
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-16 15:28 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-17 02:23 ` Re: Adding OLD/NEW support to RETURNING Richard Guo <[email protected]>
@ 2025-01-17 10:44 ` Dean Rasheed <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Dean Rasheed @ 2025-01-17 10:44 UTC (permalink / raw)
To: Richard Guo <[email protected]>; +Cc: jian he <[email protected]>; Robert Treat <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Fri, 17 Jan 2025 at 02:24, Richard Guo <[email protected]> wrote:
>
> It seems that adding ParseNamespaceItems for the OLD or NEW aliases
> may confuse scanNameSpaceForRelid() when searching the table namespace
> for a relation item. Since they contain the same RTE,
> scanNameSpaceForRelid() might mistakenly detect multiple matches.
>
> create table t (a int, b int);
>
> update public.t set a = 1 returning public.t.b;
> ERROR: table reference 46337 is ambiguous
> LINE 1: update public.t set a = 1 returning public.t.b;
> ^
Thanks. I hadn't tested qualified table names in the RETURNING list.
I've pushed a fix for that.
Regards,
Dean
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-16 15:28 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2025-06-25 13:41 ` Dean Rasheed <[email protected]>
2025-06-26 03:04 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
1 sibling, 1 reply; 87+ messages in thread
From: Dean Rasheed @ 2025-06-25 13:41 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Robert Treat <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Thu, 16 Jan 2025 at 15:28, Dean Rasheed <[email protected]> wrote:
>
> I went over this again in detail and didn't find any problems, so I
> have committed it. Thanks for all the review comments.
>
Looking at the doc pages for UPDATE and MERGE, I realise that I missed
a paragraph in the "Description" section that needs updating.
Patch attached.
Regards,
Dean
Attachments:
[text/x-patch] returning-old-new-doc-fix.patch (1.9K, ../../CAEZATCUcqADJuapZSjPf2b6hFJ6AGOUwefRvh8Ht3UZoqqw69Q@mail.gmail.com/2-returning-old-new-doc-fix.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/merge.sgml b/doc/src/sgml/ref/merge.sgml
new file mode 100644
index ecbcd83..e3d9ca6
--- a/doc/src/sgml/ref/merge.sgml
+++ b/doc/src/sgml/ref/merge.sgml
@@ -106,10 +106,11 @@ DELETE
to compute and return value(s) based on each row inserted, updated, or
deleted. Any expression using the source or target table's columns, or
the <link linkend="merge-action"><function>merge_action()</function></link>
- function can be computed. When an <command>INSERT</command> or
+ function can be computed. By default, when an <command>INSERT</command> or
<command>UPDATE</command> action is performed, the new values of the target
- table's columns are used. When a <command>DELETE</command> is performed,
- the old values of the target table's columns are used. The syntax of the
+ table's columns are used, and when a <command>DELETE</command> is performed,
+ the old values of the target table's columns are used, but is it also
+ possible to explicity request old and new values. The syntax of the
<literal>RETURNING</literal> list is identical to that of the output list
of <command>SELECT</command>.
</para>
diff --git a/doc/src/sgml/ref/update.sgml b/doc/src/sgml/ref/update.sgml
new file mode 100644
index 12ec5ba..40cca06
--- a/doc/src/sgml/ref/update.sgml
+++ b/doc/src/sgml/ref/update.sgml
@@ -57,7 +57,8 @@ UPDATE [ ONLY ] <replaceable class="para
to compute and return value(s) based on each row actually updated.
Any expression using the table's columns, and/or columns of other
tables mentioned in <literal>FROM</literal>, can be computed.
- The new (post-update) values of the table's columns are used.
+ By default, the new (post-update) values of the table's columns are used,
+ but it is also possible to request the old (pre-update) values.
The syntax of the <literal>RETURNING</literal> list is identical to that of the
output list of <command>SELECT</command>.
</para>
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-16 15:28 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-06-25 13:41 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
@ 2025-06-26 03:04 ` Robert Treat <[email protected]>
2025-06-26 09:23 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
0 siblings, 1 reply; 87+ messages in thread
From: Robert Treat @ 2025-06-26 03:04 UTC (permalink / raw)
To: Dean Rasheed <[email protected]>; +Cc: jian he <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Wed, Jun 25, 2025 at 7:42 AM Dean Rasheed <[email protected]> wrote:
>
> On Thu, 16 Jan 2025 at 15:28, Dean Rasheed <[email protected]> wrote:
> >
> > I went over this again in detail and didn't find any problems, so I
> > have committed it. Thanks for all the review comments.
> >
>
> Looking at the doc pages for UPDATE and MERGE, I realise that I missed
> a paragraph in the "Description" section that needs updating.
>
> Patch attached.
At first look this seems right, modulo some typos
+ the old values of the target table's columns are used, but is it also
+ possible to explicity request old and new values. The syntax of the
should be "but it is also" and "explicitly".
Robert Treat
https://xzilla.net
^ permalink raw reply [nested|flat] 87+ messages in thread
* Re: Adding OLD/NEW support to RETURNING
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-16 15:28 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-06-25 13:41 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-06-26 03:04 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
@ 2025-06-26 09:23 ` Dean Rasheed <[email protected]>
0 siblings, 0 replies; 87+ messages in thread
From: Dean Rasheed @ 2025-06-26 09:23 UTC (permalink / raw)
To: Robert Treat <[email protected]>; +Cc: jian he <[email protected]>; Jeff Davis <[email protected]>; PostgreSQL Hackers <[email protected]>; Tomas Vondra <[email protected]>
On Thu, 26 Jun 2025 at 04:04, Robert Treat <[email protected]> wrote:
>
> At first look this seems right, modulo some typos
Oops, yes that was careless. Thanks for checking. I've fixed those and
pushed it.
Regards,
Dean
^ permalink raw reply [nested|flat] 87+ messages in thread
end of thread, other threads:[~2025-06-26 09:23 UTC | newest]
Thread overview: 87+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH v19 1/3] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2019-08-09 17:23 [PATCH v4] Rejigger code Alvaro Herrera <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v31 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v28 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v23 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v32 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v26 04/10] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v25 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2022-03-23 18:50 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2022-03-29 12:34 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2022-08-02 18:56 ` Re: speed up a logical replica setup Jacob Champion <[email protected]>
2024-01-08 07:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 09:27 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-25 10:16 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 13:03 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-18 09:43 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-19 08:48 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-25 05:54 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-26 00:28 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 05:51 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-29 09:57 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 15:37 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 17:39 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-23 23:58 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-10-14 11:03 Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-14 15:34 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2024-10-14 16:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-10-29 13:05 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2024-11-28 11:45 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-01 08:19 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-03 19:39 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-01-07 10:56 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-08 06:16 ` Re: Adding OLD/NEW support to RETURNING jian he <[email protected]>
2025-01-08 09:38 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-16 15:28 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-01-17 02:23 ` Re: Adding OLD/NEW support to RETURNING Richard Guo <[email protected]>
2025-01-17 10:44 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-06-25 13:41 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
2025-06-26 03:04 ` Re: Adding OLD/NEW support to RETURNING Robert Treat <[email protected]>
2025-06-26 09:23 ` Re: Adding OLD/NEW support to RETURNING Dean Rasheed <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox